Government Statistical System in India-Management Challenges and Opportunities

(This report, comprising five chapters and Bibliography, was submitted in February 2012 at IIM, Lucknow by my fellow ISS officers, S/ Shri Satyabrata Chakrabarti, Mukat Singh, Tapas Kumar Sanyal, and myself, as a requirement under Advanced Management Programme.)

1.1     We are in the era of ‘knowledge economy’. It is the latest stage of development in global economic restructuring. The world economic order has thus far moved from an agricultural economy (pre-Industrial ‘agrarian’ Age) to industrial economy (with the Industrial ‘manufacturing’ Age) to post-industrial ‘service centric’ economy (mid-1900s) to knowledge economy (late 1900s – 2000s, largely the technology/ human capital sector). This latest stage is characterized by the proliferation of technological innovations and the globally competitive need for innovation with new products and processes that develop from the research community. Statistics being products of systemic processes of technological treatments and contributing to knowledge bank are also integral neural part of knowledge economy. As the largest statistical system, the government statistical system is very vital for knowledge-based economy. In the Indian context, its potential is not only enormous but also highly under-valued.

1.2     The Objective of this project is to explore and find out as to how the system has evolved and in the process has developed certain fundamental problems that are typically management disorders and potentially rectifiable with changes in some of the institutions (principles, rules, codes besides structural aspects) of the system with more clear and defined management re-orientation of statistics managers. With the purpose of making the analysis and findings of the work useful to the managers, it also provides an overview of the challenges involved and the opportunities in the context of current state technological development and knowledge management.

1.3     Known for its fundamental strengths in structural configuration, operational vastness and in-built skills, Government Statistical System of India is all about Statistics produced and used in the government at all levels and for users beyond the control of the government. The system, by and large, follows the constitutional framework of division of powers between the centre, states and sub-state level institutions. The peculiar feature of the system is that it is decentralized in respect of production as well as use in the concerned government offices. No office in the government produces statistics solely for its own consumption and no office produces all the statistics that is required or used by it. This creates inter-dependence both vertically and horizontally in the government set-up across different organs of the statistical system. Secondly the statistics produced more often than not reflect directly or indirectly on the performance of the government of the day, which requires the freedom for the production process from undue influences. Thirdly, the agencies at the centre and in the states that are required to provide leadership in maintaining statistical standards and protocols being also engaged in production have not been able to exercise their leadership role.  Even in cases where main professional functions have been assigned to statistical organizations at the Centre and in the states, the administrative statistics being in the domain of the concerned line ministries, a kind of compartmentalization has automatically crept in into the system. These issues coupled with the problem of different cadres of personnel working in different offices under a, by and large, disconnected system have aggravated the problem of harmonization, quality, and timeliness. Lack of quality standards and protocols has contributed to lack of public trust in statistics.

1.4     This paper brings out the aforesaid issues in great detail from the angle of managing the statistical system and attempts to provide possible options to address the issues and also  develop optimal and sustainable solutions. It also attempts to take into account some of the related recommendations of NSC headed by C.R Rangarajan which comprehensively reviewed the system. The policy dimensions which needed special attention for developing an efficient statistical system in keeping with the emerging priorities and changing realities have also been dealt with.  Attempt has also been made to develop the statistical system in such a way that it becomes a public good in the sense that they are contextually relevant, technically robust and widely accessible in a cost-effective manner.

1.5     Any strong system of government’s official statistics in a country may be characterised by a few attributes viz., (i) institutionalised and professionally managed systems and sub-systems with structured protocols for vertical and horizontal controls and cohesion consistent with government’s functions at different administrative levels; (ii) rigour of principles, rationales, concepts and methodology for operations, processes and production of statistics, and streaming them for purposive end-uses; (iii) amenability to change and evolve as a major instrument of social and economic development processes; (iv) being able to maintain credibility; and (v) having high degree of freedom from political interests and motives.

1.6     The virtual extinction of local level administrative data recording system, or the highly contaminated condition in which they existed, made it impossible to get any reliable statistics on villages, blocks and districts of the country on regular basis.

1.7     Evolution of the statistical system, has been given in Chapter-2 to make us appreciate the vicissitudes that had occurred in the past through which we were led to the present context of decentralization. Institutions for Interdependence in the decentralized system are described in chapter-3.Growth and Development issues are given in some detail in chapter-4. The main management challenges and opportunities have been amplified in chapter-5, which induce integration of the system without disturbing decentralization.

Experience is not what happens to a man, it is what a man does with what happens to him. ­- Aldous Huxley

2.1     We find two types of government statistical systems in the world today: in terms of their management as unified entities within the government of a country, they are typically referred to as ‘centralised’ system and ‘decentralised’ system. India’s is unique in the sense that under the federative structure of governance in India, the statistical systems of both centralised and decentralised types have been made to co-exist. The division of these two systems has evolved from the division of administrative functions between the Government of India and the State Governments and are governed by the subject classifications under the Union, State and Concurrent Lists in the Constitution of India adopted after the independence of the country from the British rule in 1947. In the literature on official statistics of India, however, the government system of statistics has been described essentially as a decentralised one: ‘laterally decentralised among the Ministries of the Government of India (GOI), and in every one of them, vertically decentralised, between the Centre and the States’ (Report of the National statistical Commission; …….). The characterisation ascribed to the system by us here on the other hand, lies in how we would dichotomise the management contexts in the later parts of this chapter as a major discord. It is not difficult to comprehend that the Indian federal structure and the large sizes of its population and geographical area have influenced the organisation of the statistical system in this fashion. It is neither difficult to appreciate the fact that it is the legacy of the British rule in India till 1947, which contributed greatly to this constitutional position of the Indian statistical system.

2.2     Recognised as one of the better systems in terms of the properties of strength stated before, India’s statistical institutions have over the years been afflicted with discordant development of control dynamics at different levels, different places, and at different times, which could be traced in what happened during the hundred years of statistical practices under the British rule. Here by ‘statistical institutions’ we mean not only the organisations and their structural parts, but also the principles, rules, norms and methods of their governance and activities. We will look at these aspects in the following sections and try to bring forth their relevance in the current context. In understanding the management challenges and also the opportunities in the present forms of institutions of the system, a critical appreciation of some of the fundamental features of the evolution process that took place during the British regime and led the Government of India post-independence to devise a system that the then ‘context’ had strong reason to imbibe.  

Decentralisation: the British Experience

2.3     The British governance system by virtue of its systemic compulsion had laid out decentralised system of statistics making provincial governments responsible for their statistical activities which were fundamentally built around administrative processes and were predominantly registry based. We essentially mean by this that there was hardly any application of sample survey techniques for production of statistics; nevertheless, long before India’s formal integration with the imperial Government of Britain in 1861, the East India Company badly felt the need of a statistical survey of the country way back in 1807, with a view to rationalize and establish a sound system of revenue collection, for its limited objectives.

2.4     The real form of decentralisation of the British Imperial governance across the country after 1861 had in fact led to a prompt review of the decentralised statistical functions already vested in the provincial governments. By 1861, the Provincial Governments of British India were already mandated to publish the relevant statistics in their annual administration reports. And for this purpose, they had the district offices to depend upon for statistical inputs related to respective districts covering a wide range of subjects. In the process of evolution of the statistical system during these early years there were some important milestones that may be worth citing. First thing is about recognising the need to create an overall view for the central government. Towards amalgamation of provincial statistics into one national compendium, the first significant development was the constitution of a Statistical Committee (1862) for the preparation of forms to collect statistical information on different subject areas. This is the first recognition of the fact that uniformity of data elements and their inherent characteristics are fundamental to their upward aggregation, comparability over space and time and for meaningful analysis of situations at different levels of government administration. This led to the publication entitled Statistical Abstract of British India in 1868. This publication was based on the returns of the local administrations and contained useful statistical information for all the British Provinces, and became an annual feature till 1923. In almost similar fashion, following the recommendations of the Indian Famine Commission, Agriculture Departments were opened in 1881 in various provinces inter alia for collection of Agricultural Statistics, under the coordination in the collection of Agricultural Statistics by the Provinces vested in the Department of Agriculture. The first publication on the subject, Agricultural Statistics of British India, was brought out in 1886. The Indian Industrial Commission (1916-1918) recommended that a Department of Industries should be created in the Provinces, with representatives throughout the Province to collect information on industries.  These events and the processes that followed them signify two policy outcomes: (i) decentralisation of the statistical work was inevitable for better management of the provincial functions, for which engaging official procedures of the administrative organs of the government at different levels was best suited and economic; and (ii) it was better policy to have national aggregates of the statistical results of the provinces for exercising central control on the desired outcomes of governance. During the years 1868-1923, the British government followed this policy before discovering the inherent problem of the policy that led to discontinuing compilation of Statistical Abstract of British India in 1923. Reflections on the problems can be traced in the reports of Commissions/ Committees appointed by the British Government on some of the other major statistical exercises as well.

2.5     The Royal Commission on Agriculture in India (1924-1925) pointed out that not only should the Provinces be self-sufficient in the field of statistics, but also that there should be a large Central Organisation. Around the same time, in 1925, the Economic Enquiry Committee was set-up to enquire into ‘the question of adequacy of the statistical data available and the desirability and possibility of supplementing it, and of undertaking an economic enquiry’. The Committee recommended that the Central and Provincial Governments should come under the supervision of one central authority that would act as the adviser to the Government in all statistical matters. The Committee supported the placing of the entire statistical organisation on a statutory basis by enacting a Census and Statistics Act and recommended the establishment of a Central Statistical Bureau, along with similar Provincial Statistical Bureaux, whose “aim was to provide a common purpose and a central thinking office on the subject of Statistics”. The British Government of India did not however, accept these recommendations at that point of time.

2.6     What the decentralisation regime of the British Government clearly exhibited was paradigmatic evidence in favour of a two-fold code:

• statistical aggregation for national level measures is meaningless unless constituent data elements are measured with better control on quality in terms of definition, scope, coverage and reference time and reference population; and
• a common standard for all aspects of data – methodological or conceptual; spatial or temporal, have to be developed by a  central authority and imposed uniformly on all subordinate systems under legal backing. 

Lessons from the Episodes of Crisis

2.7     Episodic developments some time leave long lasting effects on evolution of statistical practices and institutions. We have witnessed this towards the end of the British era, when certain big events kept the administration at bay. The economic recession of the early thirties was the first of the major shocks. The Committee appointed by the Government of India in 1934 under Messrs. Bowley and Robertson, for facilitating a further study of economic problems in India, was required, according to one of its terms of reference, to make recommendation about the organisation of a Central Statistical Department. The two experts were clear that there should be in each major Province a whole time Statistician who would cooperate with the Central Director of Statistics and who would be as nearly independent of departmental control as administrative requirements permitted. The creation of the Central Economic Intelligence Organisation, under the Economic Adviser, and the emergence of the Department of Commercial Intelligence and Statistics were the results of these recommendations. Similar developments followed in the Provinces and the United Provinces Government was the first to set up a Department of Economics and Statistics in 1942. The Government of Bombay followed by the establishment of its Bureau of Economics and Statistics in 1946.

2.8     The outbreak of the War in 1939 was another big event which gave a fillip to the development of statistics to meet the requirements of the Government in the years to follow. In 1945, the Government of India set up an Inter-Departmental Committee headed by the Economic Adviser to the Government of India to consider the statistical material available and to make recommendations for filling up of the gaps, and for improvement in the existing organisations. Among the organisational recommendations was a scheme coupled with the formation of a Central Statistical Office for coordination, the institution of a statistical cadre, establishment of Statistical Bureaus at the Headquarters of State Governments and the preparation of overall statistics for the entire country.  These recommendations as outcome of the British concern for wartime exigencies turned out to be the foundation stone of government statistical system post-independence.

2.9     In the mean time following the famine of 1942, the government set up another powerful commission: the Famine Enquiry Commission (1945), which went on to suggest the appointment of qualified Statistical Officers at Provincial Headquarters to assist the Director of Agriculture.

2.10   These episodic developments set the ball rolling towards institutionalisation of a new order of statistical practices- a paradigm shift from a purebred decentralised system to a centrally controlled system with defined institutions (organisational structures, methods, standards and referential framework) for the common national purposes. The post-independence regime for statistical operations was developed drawing on the same model, with only one fundamental difference, which was the consequence of what Prof. Mahalanobis conceived of the system –a strong centrally controlled network for regular year-to-year nation-wide sample survey operations. This was the result of recognising for the first time the fallibility of central control on decentralised administrative statistics.

Eroding Administrative Statistics and the Conflict

2.11   Post-independence era in the evolution India’s statistical system is marked by many laudable developments, thanks to great visionaries like Prof. P.C. Mahalanobis and the scholastic contributions in the fields of theoretical and applied statistics made by many Indian statisticians. Behind the backdrop of these developments, the administrative statistics started losing the favour of professional statisticians and the control on technical areas was diminishing under the newly found statehood and democratic polity. The erosion was insipid and not visible as a major disorder till very recent times.

2.12   The coming of the era of developmental planning in India, for which Prof. Mahalanobis master-minded the systemic institutions of official statistics, gave significant impetus to the development of statistics as an official system of professional engagement with statistical paradigms. In the process, the required integration was happening around the institutional centrality of control norms, methods, networked operations and resource distribution. These are clear from the few important milestones of institutional development that took place post-independence as detailed below (http://www.mospi.gov.in/……):

2.13   These developments were essentially meant to provide the overarching institutionalisation  for the symbiosis of those parts of the system that existed long before the State Directorates of Economics and Statistics and the Central Statistical Organisation (CSO) at the Centre were created, and which exists even now, together with the modern ones. Thus a combination of decentralised and centralised systems came into being. One was built upwards from district offices of formerly Provincial and now State Government departments, to the level of these departments, and from there to the corresponding ministries at the Centre for administrative statistics while the latter emerged as CSO/ NSSO centric ‘non-administrative’ brand of statistics under management and operations by professional statisticians of the Central Ministry (later on known as MoSPI) spread across the country. This is not truly a collection of State-level systems forming a National system. On the one hand, the system turned out to be, as we have described earlier, laterally decentralised among the Ministries of the Government of India (GOI), and in every one of them, vertically decentralised, between the Centre and the States, and on the other hand there is (MoSPI-run) centralised system for collection, and compilation national and state-level (non-administrative) statistics as per central mandate and with the help of statistical personnel of MoSPI located at the central and state/ regional level offices.   

2.14   The bond between the State departments and the Central Ministries in the field of administrative statistics was some what strong under the British regime, mainly because of the unified rules of administration under one government at every level. Though the British Government also realised the need to have better control over decentralised entities of statistical functions in terms of conceptual and methodological standardisation through establishment of a Central Statistical Bureau, along with similar Provincial Statistical Bureaux, it could not enact the legal statute for that, mainly because of its oneness of ruling all parts of the government and common objectives at central and provincial levels. After independence, compelled by the principles of federative governance, the statutory instruments (the Collection of Statistics Act, 1953 etc.) were put in place to enforce objectivity, rationale and rigour in the system together with executive cutting edge for the statistical operations. Progressively in course of time, as divergent political agendas emerged in the States and at the Centre through democratic processes, the political compulsions and priorities of the government at the Centre and those in the States and their consequential influence on the systemic entities of statistical functions have been frequently at conflict. With the introduction of more and more number of development programmes by the Central ministries, the statistical functions entrusted with grass-root level functionaries were getting neglected, the recording/ registry system virtually collapsed and the sub-national statistical systems had fallen prey to non-professional and mis-managed processes, which badly reflected on the quality, timeliness and coherence of the statistical returns of the State departments. The management inefficacy was so pervasive that hardly any mechanism could stem the rot right from the very roots namely, the very first stage of collection and recording of data, particularly in sectors like: agriculture, labour, industry and commerce, through all other stages of aggregation. The CSO’s modus-operandi to address the crises fell short of expected results. The CSO carried out its function of lateral coordination in statistics between it and other ministries mainly by means of the technical committees or working groups, appointed for specific purposes. The other mode was the bi-annual Conference of the Central and State Statistical Organisations (COCSSO) organised by the CSO. The COCSSO provided a forum for exchange of views and experiences concerning development of statistical activities in the country. The CSO and NSSO line of statistical rigour and professionalism got restricted within their own realm of statistics production and failed to impact the processes in other ministries mainly for coordination barriers erected on the channels of communication and interdependence in performing complementary roles on mutually set agenda based on cross-pollination of ideas through extensive discourses among the stakeholders and users of statistics.  

2.15   The diagnosis of the discord leading to breakdown of interdependence in the words of the Rangarajan Commission (2001) as follows did spell out a few managerial dimensions of the problems:

‘The success of the CSO in its role as a co-ordinator depended, on the one hand, upon the degree of its initiative and ability to persuade, and on the other hand, on the co-operation of the ministries, and their willingness to participate in this process as a team and to be persuaded to accept the conclusions of the team about their statistical work. However, given the historical background, the statisticians from the ministry had a less flexible mode of thinking, being generally averse to change, and “outside” influence.’ (The Report of the National Statistical Commission constituted by the Government of India under the Chairmanship of Dr. C. Rangarajan for reviewing the statistical system of the country submitted its report to the Government in the year 2001.)

2.16   The key words of the above statement and associated cause of dis-functionality attributed to the managers required special focus:

1. lack of initiative ( was there no ‘stimulus’?)
2. inability to persuade (to what extent incompetence, lack of conviction or lack of communication skill were barriers?)
3. (attracting the ministries’) cooperation and willingness to participate  in the statistical processes (was the conflict due to failure to recognise the complementary roles of CSO and other ministries or, non-acceptance of professional and authoritarian functions of CSO/ NSSO in the technical parlance?)
4. (non-acceptance of) conclusions of the team ( was the circle of influence for management confined to one or a few senior managers only?)
5. less flexible mode of thinking of the statisticians  (could it be attributed to the historical background alone?)
6. averse to change (zone of comfort or insensitivity to users’ need?)
7. averse to “outside” influence (constricted space for development or indifference to learning ?)

2.17   Critical components of these attribution which are on the statisticians in their role to be able to perform as better managers of situations leading to discordant relationship between the centralised stream and decentralised stream of statistical activities are lack of initiative, failure of persuasive communication, inflexible mindset, aversion to change and inertness to outside influence. The net effect of a successful management of the affairs desired to be achieved at the end was building mutually rewarding relationship.

There can be no friendship without confidence and no confidence without integrity. – Samuel Johnson

3.1     In the preceding chapter we have observed how the CSO/ NSSO run centralised system and any administrative ministry run decentralised system failed to establish interdependence between each other. The consequence of this discord is visible in mainly two different ways: (i) that the statistical products coming out from each of the two streams failed to match the expectations of either one in terms of scope, coverage or timeliness, even at times the credibility; and (ii) that there was great deficit of public trust and lack of confidence in the minds of users about the system in general. We have also observed in the previous chapter that the Rangarajan Commission attempted to identify reasons for this discord and attributed it to the failure of ‘communication with persuasiveness and sincerity’ of the statisticians in the administrative ministries and the CSO/ NSSO. However, the managers of mainstream statistical system (i.e., the CSO/ NSSO run centralised system, the decentralised stream being in the role of playing the second fiddle) were prudent enough to realise that beyond the personal skills (communication skills, initiatives, etc) of the statisticians, the institutional aspects in the recommendations of the Rangarajan Commission should take the driver’s seat and in turn bring about changes in the system as well as in the management skills of the statisticians. The actions taken by the Government of India include institutionalising a permanent National Statistical Commission (NSC), defining its functions and powers and also creating the National Statistical Organisation (NSO) under the MoSPI by redefining the Central Statistical Organisation as the Central Statistics Office, the National Sample Survey Organisation as the National Sample Survey Office and the Computer Centre which form the functional entities of the NSO for better cohesion among these organs of MoSPI to perform, beyond their statistical exercises, the NSO’s role to provide a common purpose and a central thinking office on the subject of Statistics.  By these institutional changes, which were intended to encompass the principles, rules, norms and methods of the statistical activities apart from structural re-organisation and strengthening, a new regime of interdependence came into being since the year 2005-06.

3.2     One of the principal reasons behind constricted inter-dependability has been the ineffective communication within and without the organisations of mainstream statistics – across the administrative ministries. The basic virtues underlying any effective communication towards interdependence within and without organisations are ‘credibility of processes and products’ and ‘confidence on and respect for institutions at both ends’. The institution of NSC getting the statutory status as the Rangarajan Commission (ibid) recommended is essentially intended to authorise the body to make just, fair, impartial and independent judgement on the statistical activities resulting in information and knowledge products as public goods. Thus in the role of a regulator, it is the ‘umpiring’ by the NSC, which is crucial for maintaining order and fair play in the game of statistical practices within NSO and beyond. It may be worthwhile to note in this context how the Menon Committee (The National Statistical Commission (NSC) vide an order dated 8^th November 2010 constituted a committee under the chairmanship of Prof. N. R. Madhava Menon to look into legislative measures on statistical matters, including a Bill to provide statutory status to the NSC. The Committee submitted its report to the NSC on 27^th October 2011.) has recently described the characteristics that the NSC should imbibe as a regulator:

‘The statutory body must be a part of “civil society”, independent of and distinct from the Government. It must play host to informed debate, provide space for alternative and dissenting viewpoints, be a voice for the voiceless, and give substance to the phrase ”participatory democracy”. Since, it must be completely independent, its structure, financing and personnel; policies should be such that it must not be at the mercy of, or amenable to pressures from the government of the day. A clear understanding of its own accountability and a good organisation structure can help to facilitate this objective’. (Report of the Committee on Legislative Measures in Statistical Matters: National Statistical Commission Secretariat, New Delhi; October 2011, page-….; Para-9.1.3)

3.3     In the government system of statistics, the functional principles, rationales, rules and operative guidelines are to be at the core of the processes and authority. Without the integrity, based on rationales and principles, the interactions between NSO and other Ministries and among all stakeholders can not effectively lead to interdependence. The NSC should be carefully vigilant and purposefully proactive to see that the professional skills and methodological inputs used by statistics managers of the NSO and other ministries during their interactions are driven by the integrity of the institutions. Covey (2004) has emphatically stressed on the importance of ‘self character’ rather than the character of others in defining the paradigm of interdependence:

‘The techniques and skills that really make a difference in human interaction are the ones that almost naturally flow from a truly independent character. So the place to begin building any relationship is inside ourselves, inside our Circle of Influence, our own character. As we become independent – proactive, centred in correct principles, value driven and able to organise and execute around the priorities in our life with integrity – we then can choose to become interdependent – capable of building rich, enduring, highly productive relationship with other people.’ (Covey, Stephen R., The 7 Habits of Highly Effective People; Simon & Schuster UK Ltd; 2004; page 187)

3.4     The relationship between the NSC and the NSO, and the principled interdependence between them are what define the internal character and circle of influence of the flag-bearing statistical agency (NSO), the agency of leadership and change, in the entire government system of statistics. Any relational disorder between the two is a potential source of trouble in the relations that the NSO needs to foster with other organisations (ministries/ departments of central and state governments, etc) of the system. The harmonisation of existing protocols of the NSC and the NSO through statutory provisions of the constitution of the NSC is one of the major challenges as there are divergent opinions about the statutory roles of the NSC and its relationship with the NSO. The Menon Committee (ibid) framework provides an opportunity to focus on the core issues of contention to be addressed. Extensive and vigorous discussions on the report of the Committee should take place within the NSO to inter-alia work out a harmonious protocol of NSC-NSO relation (and for that matter of NSC-government agencies in general). While the vexed question of authority of the NSC as regulator dissipate a sense of discomfort in the sphere of interdependence of the NSC and the NSO, the managers of statistics in the NSO need to bask in the comfort of a truly independent NSC.

3.5     The character and integrity of the NSO in the role to effectively engage its resources in execution of statistical activities within itself and interdependently by the administrative ministries have to be glaringly evident. The brand stamp of NSO may epitomize this, provided the policy instruments and the mechanisms which facilitate making the statistical products meet the standards and credible to users. For this role, the NSO can be viewed as one single person: a good manager who has to execute successfully. Its position in the system may be less fretted in relation to other partners in the business whenever it is guided by its leader: the NSC that makes the best choices for it and takes it to the right direction. In the words of both Peter Drucker and Warren Bennis, “Management is doing things right; leadership is doing the right things”. If we want the leader-manager relationship between the NSC and the NSO be effectively working, the judgement about ‘doing the right things’ will have to be founded on the principles, codes and standards. The leadership role remains always questionable unless it is rooted in firm principles, policy and codes. It has nothing to do with methods and strategies – they are instruments of the manager. At the moment there is no ‘national policy for government statistics’ and no ‘codes for statistical practices’. Without the aid of these instruments, the leadership by the NSC in regulating the direction of statistical activities would always be confronted with disrespect and disdain. Not just like the bed rock that provides strength to the system, these instruments must be transparently visible under every action of the manager (NSO) like rock beds under clear water. When the relationship of interdependence is effectively established between the NSC and the NSO, it would be much easier for statistics managers to effectively interact with other players in the game outside the NSO and establish cordial interdependence.

3.6     Statistics produced in the Government are basically of four types. The first one is statistics produced in the process of administering a law or a regulation or a rule. These are mainly the information obtained from or in respect of a class of respondents in a prescribed format with pre-specified periodicity. These are referred to as ‘administrative statistics’. This is the cheapest method of collecting statistics, as it is done as a by-product of administration through the concerned staff, who in most of the cases are not properly trained in statistics.

3.7     The second category is ‘census statistics’. The data collected periodically through human population census and other censuses fall under this category. This is the costliest method of data collection, as it requires enormous resources. Hence, a Census is normally conducted once in 10 years or 5 years depending on the need and availability of resources. It is not possible to collect all the required data through a census, as so much time cannot be spent in the field by data collectors, due to a variety of reasons such as respondents’ burden and lack of skills to collect elaborate data.

3.8     The third category called sample survey statistics fills the gap in the first two categories, as its scope is much wider and all the required data can be collected from a few respondents who can be selected statistically to represent the entire class of respondents. This work is carried out by well-trained data collectors. However, the first two categories of statistics provide professional inputs to collect data of the third category. The basic philosophy in statistics is to identify homogenous classes of respondents based on some criteria (called ancillary information) which is relevant or related to the characteristics to be studied. A sample of respondents is selected statistically from each homogenous class. The administrative statistics provides list of respondents (called list frame) with some ancillary information. The census also can provide similar information, but in many cases list of respondents is not compiled in censuses. Normally, information compiled at different levels of disaggregation (usually geographical units) in a census (called compiled frame or aerial frame) is taken as ancillary information for sample surveys. If a sample survey is to be effective, the relevant frames and ancillary information must be qualitative.

3.9     The fourth category of statistics is case studies or evaluation studies, which are normally area specific or programme specific. These are intended to evaluate specific developmental schemes of the Government, in terms of their impact towards the target populations.

3.10   The Statistical system has to produce statistics, which, more often than not, reflect on the performance of the Government. The socio-political environment in the Government and the enormity of statistics produced in different offices in the Government gave rise to many management issues. The issues range from setting priorities or goals, formulating plans and methodologies to achieve the goals, executing the plans, producing statistics of desired quality in time and disseminate them.

3.11   There are over 1200 offices in the Government at the Centre and in the States who produce statistics. About 20,000 statistical personnel work in these offices to handle statistical matters on a regular basis. If the personnel engaged for specific statistical jobs through deployment from other offices/ organizations in the Government or through outsourcing is also considered, the figure may even go up to more than one lakh per annum. The personnel work under different command and control regimes at the Centre and in the States.

3.12   Although, a few statistical subjects fall under the Union List, the production of statistics and in a few cases, even the dissemination of statistics is carried out at State level. Data collection, being a field activity, and the fact that the Centre does not have resources to collect data on its own in most of the cases makes the statistical system a decentralized system.

3.13   The peculiar feature of the system is that it is decentralized both in production as well as use of statistics among the various government offices. No office in the government produces statistics solely for its own consumption and no office produce all the statistics that are required or used by it. Thus, Government is the producer of official statistics and it is also the biggest user. There are also a few statistical offices at the Centre and in the States, who produce statistics solely for the use of other offices. Some of the offices require assistance and inputs from other offices not only in producing statistics but also in the use of statistics. All these factors created inter-dependencies both vertically and horizontally in the Government set-up across different organs of the statistical system.

3.14   Another important feature that could be noticed across different offices is that functions involving production of statistics are combined with related advisory functions or at times even with functions relating to other areas such as planning or programme implementation. In many situations, the staff units for these functions are inseparable, in a manner that one cannot identify as to which staff component is exclusively involved in functions involving production of statistics or in the other functions. In a few cases, where the functions are separable, the persons handling those functions belong to the same cadre and hence, there is a system of vertical or horizontal mobility for the persons of the cadre across the functions, thus making available a choice or chance for them to handle different functions.

3.15   For the purpose of ensuring cooperation and coordination among statistical offices, one of the offices, namely, the National Statistical Organisation (NSO) in the Ministry of Statistics and Programme Implementation (MoSPI), has been identified as the nodal office at the Centre, and one office, namely the Directorate of Economics and Statistics (DES), in each State/ UT has been declared as nodal office. The DESs are not separate Departments, but they work as sub-ordinate offices to one of the Departments (say, Planning) at the State level.

3.16   Here, we conclude this chapter with an attempt to visualize the core values in the context of the institutional changes and codes of practice which we are given at the end here by an expert Committee as key to effective interdependence.

4.2     The organisational growth curve and development curve generally maintains a gap. This gap is inevitable as development is generally slower and takes time to match the growth. When the two curves meet it may be referred to as using the full capacity or potential of the organisation.

4.3     The growth and development gap increases with time up to a point of time when growth starts plateauing and the development curve starts coming nearer to it. After some time they are likely meet. In the context of the statistical system, the development process has been nearly matching with the growth for a long period of time during which the growth has been as slow as the development. The growth started picking up much later than when the development curve had already out-stepped it. For quite a long time at this point onward the development remained constricted due to space deficit within mainstream statistics in terms of paucity of physical resources, bottlenecks in career movement of personnel, lack of amenities and infrastructure and shrinking roles for statisticians. This stage at the organisation level also produced a stunting effect on the statisticians. Gopalakrishnan (2009) cited the example of stunted crocodiles to explain how managers can also be mentally stunted. The growth of a crocodile can be stunted by confining the animal to a small enclosure soon after it emerged from the egg. He argues that similar to physical stunting, mental stunting can also happen to human beings under certain circumstances. He extends the analogy of constricted physical space to the constricted office environment and role that can cause mental stunting.

‘If the space in which the manager operates and grows is limited, if his emotional and mental exertion are low, then his development gets stunted. If he stays in this stultifying situation for long and does nothing to change his circumstance, then he can become a permanently stunted manager.’ (Gopalakrishnan, R.; The Case of the Bonsai Manager, Penguin Books India,2009; page-58)

4.4     Changes in the growth pattern in terms of physical and mental space for statisticians in government organisations started happening from mid-nineties thanks to reform measures initiated by the government, expansion of IT based operations and systemic changes in HR policies. However, the constricted space for statisticians had already generated a developmental crisis among many statisticians, of which we have already spoken citing Rangarajan Commission (ibid) that related the maladies of lack of initiative, failure of persuasive communication, inflexible mindset, aversion to change and inertness to outside influence to breakdown of interdependence (ref. chapter-2).

4.5     One reason for the lack of initiative on the part of statistics managers is the lack of stimulus offered by the working environment of the manager. Very often the environment is dull and devoid of challenges. This creates inertia against moving forward to test one’s potential and perform for the satisfaction of one’s internal accomplishment desire. On the other side, the work in statistical practices often had high possibility of attracting aggressive reaction from seniors, government and other external stakeholders/users. Debates by politicians at times may unsettle statistical offices and put them in great stress. Criticism by other users which attract the media attention also has the same impact, particularly when the users do not understand the fitness of statistics for use in the areas they use. These reactions tend to place the statistician in insecure circumstances. His behaviour, in turn, gets adversely affected switching off ability of persuasive communication in interactions. As a result, the purpose of having one professional statistics manager in the role of a relation builder got defeated. The statistical personnel at all levels in the Government need to follow a professional approach, no matter how they have to confront situations where their work has a direct or indirect bearing on the performance of the Government. This becomes very difficult in the context of their working under different command and control regimes and in the absence of any protection against possible victimization. The situation becomes much more awkward in the context of underlying disparities in status and salary of statistical personnel among themselves as also in comparison with other cadres. This is a kind of a situation similar to the one described in a sloka by Maharshi Valmiki, which is given below along with its meaning.

4.7     As we shift to the present context, there are a few more external challenges that the statisticians have been facing. Most of the jobs in statistical organisationsrequire research to evolve innovative approaches to improvise the system dynamically from time to time to meet changing circumstances and their internalisation. In particular, the gap between academic pursuits and official statistics has been widening, as a result of which new approaches are becoming foreign to the statistical system day by day. For example, data mining techniques being widely used by the academia and research institutions across the world have not yet made their way into the statistical system. The gap between Information and Communications Technology and its usage in the statistical system has been widening. Typically, the statistical offices do not have resources earmarked for this work even today.

4.8     Budgetary and human resource constraints are the main constraints standing in the way of growth and development of statistical offices, although in a few offices the situation is not that alarming. These constraints have implications on quality and timeliness of data that they produce and increase the effectiveness of their use. Lack of systematic study on these requirements throws up a big challenge. Having separate budgetary and human resources for statistics in offices having substantial statistical functions would facilitate an effective study of the problems and corrective measures, where necessary. The strategic intervention in the direction as follows makes management effective towards organizational development.

• rewards (as a means of motivating for knowledge sharing)
• cross-project learning
• knowledge mapping (a map of knowledge repositories within an organisation accessible by all)
• competence management (systematic evaluation and planning of competences of individual organization members)
• proximity & architecture (the physical situation of employees can be either conducive or obstructive to knowledge sharing)
• collaborative technologies (groupware, etc.)
• knowledge repositories (databases, bookmarking engines, etc.)
• measuring and reporting intellectual capital (a way of making explicit knowledge for organisations)
• social software (wikis, social bookmarking, blogs, etc.)
• Inter-project knowledge transfer

4.9     Capacity building in terms of knowledge expansion of the statistics managers, particularly in harnessing IT enabled resources is one of the challenges that the government statistical system has been facing for quite some time. The educational institutions teaching or doing research in official statistics have always been limited and the number is gradually going down. Attraction for statistical graduates and post-graduates entering into statistical profession in the Government has been showing a declining trend. Most of the offices have non-professionals handling statistical work without any training and/ or without clear directions on what is expected of them. This position is distinctly visible in offices handling administrative statistics. Due to human resource constraints in many offices, they have been outsourcing one  or the other statistical activity, which is resulting in not being able to maintain institutional memory to sustain the processes. Training has been the biggest casualty in the midst of this chaos. Learning agenda promoted by the NSO needs to foster collaborative intervention for skill development and adaptive practices with the association of research and academic institutions.

Present Context

5.1     Statistics means facts and figures and the information that emanates from it induces society to learn from the past to plan for a better future. Government Statistical System of India is all about statistics produced and used by all levels in the Government. The use of the word ‘system’ is not intended to state that everything is systematic.

5.2     There are over twenty-five federal Governments in the world and India is one of them. India is described as quasi-federal or as a federation with more unitary features. The statistical system, by and large, follows the constitutional framework of division of powers between the Centre and the States as also of sub-state level institutions. In the case of Jammu & Kashmir, the division of powers is guided by the Seventh Schedule as applicable to that State. Government statistics are also known as ‘official statistics’. These are used or purported to be used by the Government in informed decision making, planning and policy formulation. They also induce informed debate within and outside the Government.

5.4     The de-licensing and de-regulation initiated in the nineties under the Liberalisation, Privatisation and Globalisation (LPG) regime made some of the administrative statistical systems weak as they were no longer capable of collecting data on statutory basis. The regime coupled with WTO agreements gave rise to enormous increase in data needs. The Government brought up the Collection of Statistics Act, 2008 to enable meeting most of these requirements.

5.5     Internationally, many countries have made laws and re-engineered their institutions in a manner consistent with the UN Fundamental Principles of Official Statistics. The principles, if adhered to, would promote using statistics only for statistical purposes, public trust and statistics as a public good.

Main Challenges and opportunities

5.6     There may be views that it would be difficult to maintain the independence of the statistical system with too many offices involved in the system. But, this is not true in our opinion.

Interdependence is a higher value than independence. – Stephen R. Covey

5.7     It is a well accepted principle that in order to sustain growth or development, change is a necessary pre-requisite. Since large organisations react very slowly to changes in the situation, their capacity to sustain growth/ development becomes difficult. In small units, it is relatively easier, as they adopt quickly to changes. Centralisation or establishing offices of large staff components resulting in reduction of number of offices would not catch up to change quickly as is possible in case of offices of smaller and manageable sizes of staff component. For this simple reason, the decentralization and the associated inter-dependencies built in, into the system have been a virtue, but not a weakness. This also fosters competition and better environment for creativity. However, the system, in the context in which it has been functioning over the years, has thrown up a number of challenges.

5.8     Typically, the steps usually involved in the Government statistical system are as follows.

5.9     Following all the aforesaid steps systematically is by itself a challenging task to any office producing statistics.

Aristotle maintained that women have fewer teeth than men; although he was married twice, it never occurred to him to verify his statement by examining his wives’ mouths. – Bertrand Russel

5.10   A typical office in the Government is involved in production of a set of statistics year after year and its activities become so repetitive that it tends to ignore or it does not get time to follow one, or more than one, of the aforesaid steps. At times, closeness to the system also makes them blind.

Yad yad acharathi Sresthas Tat tad evetaro Janah Sa yat Pramanam Kurute Lokas tad anuvartate   “Whichever and however a great personality conducts himself common men do also; whatever he accepts as authority that and that alone certainly all the world will follow.” – Bhagavat Gita

5.11   In the case of the thirty-six nodal statistical offices, which are mandated to advise other offices in statistical matters, the expectations are relatively higher. When something goes wrong in the system, the users look forward to their response/ reaction or find them wanting in one aspect or the other. The position becomes much more awkward when a nodal office itself is involved directly or indirectly in production of statistics. Other offices view them as a leader and if there is any lack of initiative or if they do not practise protocols that ought to be practised, the execution of nodal functions becomes weak. Thus, the nodal agencies at the Centre and in the States, which are required to provide leadership in maintaining statistical standards and protocols, being also engaged in production have not been able to exercise their leadership role effectively.

5.12   Even in cases where main professional functions have been assigned to nodal statistical organizations, the administrative statistics being in the domain of the concerned line Ministries/ Departments, a kind of compartmentalization has automatically crept in, into the system. These issues coupled with the problem of different cadres of personnel working in different offices under a, by and large, disconnected system have led to lack of harmonization, quality, and timeliness. It also led to avoidable duplication and at times, conflicting statistics. Lack of transparency in statistical operations, quality standards and protocols has been the main cause of the system not being able to acquire public trust in statistics.

5.13   The gap between the Information and Communications Technology (ICT) and its usage in the statistical system has been widening. Inter-dependence in the system requires that all the offices should be connected through a network to facilitate sharing of information and better communications among them.

5.14   Some of the large organizations have not adopted themselves to the latest developments in the ICT field. While standard software packages like SPSS, STATA are available to handle some of their functions, such as drawing sample lists, estimating standard errors of estimates and report generation (tabulation), they are still going by the beaten track and using heavy amount of resources and time for carrying out these activities. Such organisation should consider using the available technology by upgrading the skills of their staff in the usage of available software, rather than in preparing software in-house. Performance management tools should be used in carrying out various survey programmes to ensure better execution of the goals set for the purpose.

5.15   There are several data sets produced in the system on a regular basis. Often, helplessness is expressed with such mountains of data. The potential of such data to aid sound policy formulation can be fully exploited by using data mining techniques. Data mining brings out unknown facts from the data. Today data mining techniques through available software like Microsoft XLMiner are being widely used in variety of disciplines. These are user-friendly software packages and can be learnt through moderate effort. The process of fitting the best model based on the pattern in a time series data, testing the efficacy of the model by comparing the actual and estimated values to the satisfaction of the decision maker and then predicting future values provide immense scope for its usability in meeting varied requirements of official statistical system and individual organizations and researchers.

It is an extraordinary thing, of course, that everybody is answering questions without knowing what the questions are. In other words, everybody is finding some remedy without knowing what the malady is. – Jawaharlal Nehru

5.16   Another area that is important to run the system effectively is the consultancy process. In particular, due to the large size of the nation with wide diversity, it has become difficult even to know the list of users. Even in cases, where the users are known, the manner in which statistics are used by them is not known, which widened the gap between policy makers/ users and statistical offices. In some cases, the policy makers require very quick products even of less quality than what is desirable, which is not appreciated or at times not in the knowledge of concerned statistical offices. This leads to wastage of resources as the offices, in pursuit of desired quality, take more time to produce such statistics, which become either unusable or outdated.

5.17   It is necessary to ensure coordination and cooperation among offices producing statistics in the following areas, which are indicative but not exhaustive.

• Avoiding unnecessary duplication
• Sharing of statistics produced with other offices
• Identifying which office is more competent and appropriate to take up a specified job
• Sharing action plans on production of statistics
• Sharing technical expertise and success stories
• Sharing experiences of using statistics, particularly in respect of quality and timeliness

5.18   There should be continuous dialogue of statisticians with policy makers and users, particularly with those who operate the administrative statistical system. In meetings, such as the Conference of Central and State Statistical Organisations (COCSSO), the agenda to be discussed shall emanate not only from the Centre but also from state level institutions. A periodical survey on the performance of statistical offices shall be conducted, say, once in five years to understand the systemic deficiencies and the issues should be addressed in the COCSSO.

5.19   If an office has been producing a specific product, which it has not been using much, the office tends to stop its production or tends to produce it without the same quality and timeliness as was the case earlier. This puts the work in other user offices in disarray. Agricultural statistics is a case in question. There could be many such cases. Such practices should be discouraged and the choice of office to carry out any work has to be done carefully, keeping this aspect in view.

5.20   Official statistics is a large domain. Managing the enormous data that the system generates year after year and maintaining confidentiality of personal information contained in the data is a huge task. Adequate safeguards have to be developed for the purpose.

5.21   Statistical products, and in particular those that are regularly or periodically produced by one office, require statistical audit through an external expertise. It may be argued that internal quality control instruments are adequate, but more often than not, these instruments address implementation processes and do not cover the entire gamut of activities, that too to the extent it would be feasible within the given time-frame and resources. Many countries are moving towards establishing quality control frameworks and statistical audit mechanisms. The National Statistical Commission has taken the initiative to get one of the products of the Central Statistics Office audited by a Consultant. Such measures need to be continued to strengthen the quality and credibility of regular/ periodical products.

5.22   In order to promote trust in official statistics among the users and the public at large, policy making in the areas of priority setting, standardization, protocols for ensuring transparency in statistical operations, proper dissemination policy, quality control framework and statistical audit, monitoring mechanism for ensuring quality, timeliness and credibility of important statistical products and facilitating innovation and research, if done and implemented in letter and spirit would meet the legitimate expectations of the public at large and the users in particular. The National Statistical Commission has initiated the process of laying down a National Policy on Official Statistics, which would set a common agenda and minimum uniformity in professional conduct for statistical offices in a decentralised setup. This process would eventually promote statistics as a public good in such a manner that they would be contextually relevant, technically robust and widely accessible in a cost-effective way.

5.23   The National Statistical Commission, on the basis of the report of a Committee appointed by it under the Chairmanship of Prof. N.R. Madhava Menon, made three important recommendations to the Government, which are briefly indicated below.

• Bringing census on any subject and statistics of national importance under the purview of the Central Government by suitably amending the Union List in the Seventh Schedule of the Constitution
• Establishing the National Statistical Commission as a public corporation through an Act of Parliament to regulate statistics of national importance and to advise on other official statistics
• Constituting an all India Statistical Service to create a network of officers to run the system

5.24   The aforesaid measures, if implemented, are expected to provide a pro-active environment to address most of the issues raised in this report. Integration of the statistical system without disturbing its decentralized structure becomes feasible through these measures. These measures would also promote unification of statistical offices and personnel for achieving national goals without causing any disturbance to decentralisation.

5.25   There could be a view that ‘official statistics’ would be dichotomized as ‘statistics of national importance’ and ‘other official statistics’, which may create insecurity and discourage creativity in the system. But, it will not be so. The good practices and protocols that may be introduced in respect of statistics of national importance would induce similar measures to be followed voluntarily in other official statistics and there will be a healthy competition between both the components of the system. Since, creativity would be open in other official statistics, innovations, if any, in this area would induce the component on statistics of national importance to follow suit. Thus, both the components would act as nudges to each other and it would foster integrated development of the overall system.

Bibliography: