Making Cities Open by Default: Lessons from Open Data Pioneers

Open data can have a really positive impact on the ways in which city and local governments operate. We heard a number of reasons why this is the case.

A key argument in favour of open data is that it has the potential to rebuild trust and integrity within government. According to a city official from Durham, North Carolina: “Trust and good faith in government go a long way. Open data is a barometer of good faith. For residents to trust government, government must trust residents.”

An element of building trust is to use open data to improve the dialogue between government and residents. This point was made by the director of Ontario’s Open Government Office, who said: “If we’re thinking about public engagement, and how we can engage differently, it starts with thinking about open data and information.”

If published and used in the right way, open data can improve the functioning of government and push expectations. For example, it can allow citizens and third parties to develop the sorts of tools that city officials either don’t have the capacity or resources to create. A city official from the City of Edmonton made this point clearly: “People work and contribute in their spare time. Analytical approaches, software skills that the government can’t afford, don’t have enough of, or haven’t even heard of, exist within the open data community. Things that weren’t within the scope of the government, now the government wants to keep up.”

Finally, open data has the capacity to help encourage sustainable development and economic growth. Open access to publicly held data can provide greater returns on initial public investment. A city official from Lviv in Ukraine made clear that this was a key part of the political sales pitch for open data: “Opening data is a part of the Mayor’s vision for economic development. We have a specific focus on how open data can be used to start enterprise and contribute to the creative economy.”

The Open Data Charter has the potential to help officials and civil society reap the benefits of open data. According to our interviewees, the advantages for a government adopting the Charter include:

• The Charter provides a global standard for government policy design in support of strong, local, open data programs.
• Adopting the Charter is a political signal that a government believes in openness and desires a connection to the broader open data community, both locally and internationally.
• Adopting the Charter provides governments with a clear set of initial actions that should be taken to ensure that the principles are being upheld and achieving the maximum benefit.
• The Charter provides a clear global standard for governments to evaluate the strength and impact of their open data program.
• The Charter connects adopting governments with their peers and colleagues around the world, providing learning and networking opportunities.

However, the Charter is not a magic bullet, and some municipalities do not see the incentive to formally adopt the Charter if their open data policies already align with the Charter principles. This echos the findings of the 2016 OpenNorth report on cities and the Charter (Bryson E., Johnson P., Landry J-N. (2016) What Could Open Data Programs Gain from Aligning with International Best Practices? (p.11) OpenNorth, https://drive.google.com/file/d/0B739vUevKlPgZEdVcFoxb0ZITmM/view). Further, some interviewees suggested that they view adoption of the Charter as an achievement of a certain status rather than a set of principles that informs their open data programs.

Open by default is the first principle of the Charter and it represents a real shift in how government operates and how it interacts with citizens (Open Data Charter principles: https://opendatacharter.net/principles/). At the moment, the public often has to ask officials for the specific information they want. Open by default changes this dynamic and suggests that there should be a presumption of open publication. If governments need to keep data closed they justify why it cannot be opened, citing security or privacy concerns. Politicians see open by default as an attractive, and even mandatory endeavour, “Open data aligns with organizational values of high transparency, this is non negotiable,” stated an elected representative overseeing the City of Montreal’s open data program.

In many jurisdictions, open data may lead to the implementation of new data management systems entirely. As explained by a city representative, “The open data program is really the only way to access data for city employees too. There was no previous data warehouse in the city.” Opening data in Canadian cities renders internal data sharing more efficient, a perception corroborated by an elected representative who stated, “By incorporating data into decision-making, you demonstrate agility in addressing very specific issues”.

However, governments shifting to open by default standards is costly and can be difficult as legacy data information systems are often badly outdated. Politicians advocating for open data often lack the level of technical knowledge required to understand the complexity of moving government to this level of openness, which causes tension with the bureaucracy. The starting point for many cities is often poor data management systems, making even internal data sharing difficult. As a local government respondent stated, “We have all the different departments here essentially operating in their own silos, even though the biggest users of open data will be internal.”

A civil society leader in Toronto put it starkly: “The systems have not been built to open data… The state of government IT is in a massive technical debt, held together by chicken wire and bubblegum”.

Despite the many challenges, cities face high levels of pressure and expectations from both the public and politicians for the opening of data. As such, public sector reforms can be perceived as slow. A municipal open data program manager stated that, “the political side has been frustrated that the public sector has been slow to catch up.” This is true in Canada and international cities. The political commitment to open by default puts pressure on city administrators and addressing internal sharing issues becomes secondary. Public administrations are caught between political expectations, public pressure and the needs within their organizations.

There were varying reasons given for internal open data management sharing challenges. A civil society leader with extensive experience growing the local open data community suggested it was predominantly an IT issue, saying, “It is important to let people know that it’s usually not people wilfully holding data, it’s the technical constraints.” IT solutions require an increase in resources and planning, but are only successful if the culture is conducive to open by default and data sharing.

However, signs of resistance to open by default and data sharing remains in cities, evidenced by a local government official statement that, “there is an established resistance to sharing and accepting the additional resource burden, people need to understand why someone would need the data.” This was viewed as a “failure of imagination” by a government open data program manager, who also noted that this “should not stop the data from being released for the public use.”

As a city official from Buenos Aires pointed out, there’s a big issue of capacity and time to do open data projects: “Many agencies say they will give us the data, and it’s never given. The work is off the side of the desk for people and we have to help them understand what it means in the city and in the government”.

One local government official described the active opposition they experienced, stating, “there are pockets of resistance where people still think it’s their data. This isn’t even about opening publicly, they won’t even share the data with other departments.” More frequently, interviewees described a passive avoidance of opening data. Data holders did not see a high demand from the public and needed to be educated on the benefits of putting in the extra effort required to release their data.

In both international and Canadian contexts, interviewees suggested that leveraging issue-based open data demands from the public was key to demonstrating the need for open data (to those resisting data sharing demand). Current policies are frequently insufficient to guide actions and lack enforceability, as expressed by a civil society leader, “The reality is the [policy] has no teeth. The policy is not applied and there is no consequence for it not being applied.” Educating data holders on the importance of open data and addressing their resource concerns is a more effective method of ensuring compliance than policy enforcement.

Local government politicians and administrators also face pressure to focus on public wins, at the expense of internal change. Participants noted that the benefits of open data as a catalyst for internal government transformation and reform were being overlooked in favour of the more visible benefits of public use (such as data visualizations and apps). There are numerous departments within municipal governments that could be working together and sharing, but currently their data is scattered and in different formats. City governments lack sufficient resources to be fully open by default; internal data sharing is one way to reduce inefficiencies and move closer to achieving full openness.

How sympathetic were civil society respondents to these expectations? A local civil society respondent said, “[Cities] need to be given the confidence to open data in the way, and at the speed, that works for their city and not have to deal with the other lobbying and pressure.” This suggests that there needs to be a shift in focus to educate politicians and the public of the gaps between their expectations and the reality of current internal open data management processes.

Those interviewed had a number of ideas for addressing these challenges. The Charter can play a role, with one civil society representative saying that, “Getting things like the ODC going speeds up the internal data sharing — staff need to be the beneficiaries and the biggest winners of an increase in data openness and sharing. It’s as simple as getting the right version of the files.”

In both international and Canadian contexts, interviewees also suggested that focusing more on issue-based demands from the public to open up data was critical to demonstrating the need for open data to those resisting data sharing demand.

Developing capacity within city governments is also crucial, with a civil society respondent saying, “training should be custom, low numbers and built around staff knowledge about why they can’t do some things. Utilize the knowledge and experience to co-develop solutions for data sharing.” Current training is still focused on the normative benefits of open data and not geared towards addressing the real life concerns of data holders struggling to adjust to open by default expectations.

Recommendations for Municipalities:

• Engage elected representatives and civil society representatives in the development of an open data strategy and operational roadmap that sets realistic expectations;
• Create opportunities for core open data teams to convene internal divisions and agencies to explain and address concerns entailed by the process of opening up data, and the ensuing organizational change;
• Bridge perception gaps through external and internal outreach;
• Work iteratively and support small, manageable pilot projects that help demonstrate the internal and public value of opening up data.

Across cities, participants reported increasing general demand for open data. Government officials feel an obligation to respond to this pressure and find ways to be more open with their data and information. An open data manager from an international city states, “government is funded by the residents so its data is property of the residents, we have a natural duty to give it back to them.”

At its best, a data literate public can provide analytical approaches and software skills that governments do not have sufficient capacity in. For a mid-size Canadian city, opening up the API for transit data resulted in an application being built by the public. This saved the city from having to procure the services to build and maintain an application. Public participation can therefore increase the scope of what is possible in civic technology, expand what the government employees know, contextualize solutions in the reality of residents, reduce the time it takes to solve problems, and reduce costs.

The pressure on cities to appear progressive and open, and release large volumes of historical and new data, has created a situation where current discourse and resource allocation is focused on data suppliers and their capacities. However, limited data literacy among the public is reducing the impact of open data on the daily lives of residents. Simply supplying open data does not necessarily result in residents being able to use the data. Respondents reported that the number of people who are data literate remain limited, suggesting that open data remains a fairly exclusive endeavour. A Buenos Aires city open data program manager stated, “We have a vocal, but small group of people asking for new data, new visualizations, and new tools. We are trying to broaden the people that want to use it [data].”

Data literacy in respondents’ cities is improving, but work remains to improve and sustain capacity building efforts. Building data literacy is a new area of focus for cities, as an elected representative stated, “In the last 8-9 years we’ve gone from zero knowledge of civic and data literacy, to being very sensitive to the issue.” The same political leader stated that, “…the data literacy of the residents is only as good as the tools you provide them, if we don’t give them the tools you don’t give them literacy.” An international interviewee stated, “we know that our mothers will not read data sets so we have to make maps and visualizations to demonstrate how the data can be shown. We have more work to do.”

There is also a risk that uneven data literacy can exacerbate socio-economic disparities, with poorer residents finding it harder to access information on city services. This is particularly true when cities rely on infomediaries to transform data into visualizations and software applications for others to use. Infomediaries do not necessarily have the same mandate as government to serve an entire population, and so may not cover poorer areas. This means that cities still have a responsibility to ensure that open data initiatives do not exacerbate existing socio-economic disparities by increasing the digital divide. Bridging the digital divide will require targeted outreach strategies in each city beyond core stakeholder communities to help communities use and transform data.

In the haste for ‘open by default,’ there has also been a focus on the amount of data opened, rather than the quality or relevance to the public. A civil society respondent reported cynicism about new releases, “most people in advocacy don’t take any releases seriously because they’ve been asking for the same data sets for 20 years and still haven’t received them.” A civil society respondent also stated that, “The goal is for residents to be able to use data in a more applied way in civic life. If governments are going to have a consultation, they should provide the necessary data to inform participants.” Open data, that is driven by user demand, is accessible and usable, can support meaningful collaboration between the government and public. People who are informed about how government works and have access to government data and information can contribute to discussions with greater depth. Releasing open data to a central portal or catalogue, without providing the public with a way to interact with government on the data, may not translate into usage and result in skepticism about the value of data release.

A Litmus Test for Public Servants

Data literacy is increasingly important in the context of smart cities where predictive data models may have a direct impact on the lives of residents, but the data and the algorithms that inform decisions are seldom available or transparent. As our interviewees suggested, there is a need for increased data literacy among the public to increase engagement in decision-making and public service delivery. Without such capacity, it will become increasingly difficult for individuals to engage and provide feedback to their city officials. Indeed, with more complexity in technologies and data processes, it is increasingly important for public officials themselves to increase their technical capacities in data collection, management, and analytics, in order to deliver on open by default commitments in smart city contexts.

From our interviews, city officials expressed a desire to understand the potential effects of their decisions and of opening up data. The overarching goal of open data needs to be clear.

“how do we help users better understand how data is used? How can the city use data to make better decisions? If we just focus on open data we miss a large component. If we’re looking at smart cities, open government is a pillar of smart or intelligent cities. We need to make it clear to the public what the end goal of open data is.” – Government respondent

Transparency and understanding of data processes, uses, and impacts is an important part of data release. A lack of capacity to understand data use can be a barrier to data usage, particularly when the data in question is managed and maintained by government. From a municipal perspective, it falls within government’s duty to open data, as government is mandated to serve citizens. An open data manager from an international jurisdiction stated, “government is funded by the residents so its data is property of the residents, we have a natural duty to give it back to them.” However, simply supplying data does not necessarily result in increased understanding of data and their context. To be transparent in data processes (and improve open data literacy), engagement with data stewards who collect and curate open data is also needed.

Public servants themselves also often need to improve their own data literacy. A number of advocates and commentators, including one of our interviewees, have recently raised the question of transparency, data governance, and government capacity in smart cities. With the onset of smart cities, public servants will be faced with increasingly vast amounts of data to process and analyse, while certain processes and decisions may be automated. In this future context, public servants will not only have to be data literate, they will need the skills and knowledge to understand how their data models, algorithms, or even ‘predictive analytics’ systems will behave with the input they provide. Without this knowledge, city officials will only be able to output the results of automated processes performed by software, and may be unable to respond to citizen feedback in an open by default manner. To ameliorate this risk, a core set of questions on technology and open data capacity can be developed to guide public officials when making procurement choices of new technologies.

(Landry J-N, Sangiambut S (2017) The Challenges to Inclusive, Open, and Smart Cities: Speed, Opacity, and Outsourcing. https://medium.com/@jeannoe/the-challenges-to-inclusive-open-and-smart-cities-speed-opacity-and-outsourcing-49ccceb45552. Accessed 27 Nov 2017)

Recommendations for Municipalities:

• Support local civic tech initiatives by participating actively in discussions about the context of data sets, past and potential uses;
• Collaboratively develop community benchmarks for the quality of data, addressing accessibility and usability by establishing inclusive local open data community advisory groups;
• Utilize existing municipal open data infrastructure (e.g. open data portal) to collect and share information about local open data community activities and impact (e.g. events, applications, data stories).
• Analyse the impacts of new technology adoption on capacity to maintain open by default commitments

Interviewees unanimously recognized the public’s expectation for governments to collaborate across jurisdictions when opening data. Our provincial representative stated, “We have been in conversation with the national government to better align key data sets and indicators across national, regional and municipal. Residents do not care who owns the data. They want access to it.”

Personal data privacy and security remain recognized issues of concern, however, city officials in Canada and internationally agreed that the general public is not concerned with how interjurisdictional sharing happens. They want broadly accessible and interoperable data and to see improvement in the services that impact their lives. Currently, innovative uses of open data are often limited by the confines of jurisdictional boundaries. The Charter is uniquely positioned to help address this challenge. As stated by a sub-national Charter adopter, “If we’re really talking about being partners with other jurisdictions, we cannot use internal policies to compel them to do anything. If we have an international body that brings others on board, it provides an instrument to work with other jurisdictions.”

Canadian municipalities, Lviv (Ukraine), and Buenos Aires (Argentina), benefit from their regional and national governments adopting the Charter and working to align the policies across levels of government. With Canada announcing its intention to adopt the Charter, and Ontario adopting the Charter, there is a unique opportunity emerging to study interjurisdictional data sharing.

Similarly, Lviv is a member of a project in the Ukraine called “Data from Cities” program in which Lviv mentors other cities that are beginning their data journeys that could serve as a global example. As municipalities design and implement their own data policies, there is room for lateral knowledge sharing networks between cities and between jurisdictions. An Canadian city official stated, “Right now we’re lifting [using] ideas from all different municipalities. We’re hoping to get caught up to the point where others are learning from us.” As cities are already informally looking to each other for best practices, there is a need for a global city network that connects open cities in the Global South and North to more systematically address their common needs and challenges.

Recommendations for national or regional governments:

• Facilitate peer learning and working groups, which leverage and support existing local initiatives and national or regional open data events to organize these activities;
• Participate in national and regional open data and open government events.

Government respondents unanimously stated that their open data policy and guidelines were created through trial and error, with attempts to learn from what other jurisdictions were doing. There is an explicit desire to learn from the precedents set by early adopters. One international city leader expressed it this way, “Other cities have more history that they can share advice on how to solve different problems. We adopted six months ago and [we are] still developing the road map. Now the real work starts – more results. This is when we need the expertise.”

Interviewees expressed that learning about the experiences of others would reduce the time it takes for them to research and develop tools and standards on their own. The Charter’s six principles provide a global framework for building a city’s open data policy, but interviewees expressed a need to apply the framework for specific strategic city planning priorities (e.g. civic innovation, resilience, smart cities). Open data policies that are only two to three years old already require significant updating. An elected representative observed, “Open data is much more complex now than it was only a few years ago. Now cities are required to address license standards, open data platforms, data collection strategies, and still monitor the political discourse.”

Several jurisdictions in this study began with an informal open data strategy and hired additional staff or contracted external support to formalize their policy in a timely fashion. Having developed policies and adopted data standards, their attention turned to implementation. A municipal open data leader observed, “Civil servants often wear two hats: the people in charge of developing the policy and fine-tuning the action plans that are now in full implementation mode.”

Key implementation challenges highlighted by open data city managers included:

• Keeping up with standards. Both Canadian and international cities want to be on the cutting edge of data standards, but open data technologies are constantly evolving, and policies and operational roadmaps are unable to keep up.
• Open data project management. When one individual is responsible for both strategy and implementation, often the strategy suffers. One civil society advocate suggested, “You can’t pull this off as a side-project without it being in specific job descriptions. There needs to be a real leaders to lead it, you can’t hive it off and make it another layer.”

Cities express a need to revisit their open data policies and identify sector specific standards. One particular challenge arose, “There is a disconnect between the needs for content areas and the technical people developing standards.” In response, adopting the Charter can serve as a driver of standards adoption as it requires the incorporation of standards into overall strategy and internal data cycle management approaches. One city interviewee suggested transit was currently the only content specific data standard being discussed in their jurisdiction. They continued by saying, “That’s the only area I know of it happening. I’m not even hearing data standards being discussed around other content areas. These are necessary, right now it’s incongruent.”

City managers are familiar with a range of standards, especially for geospatial data, but the proliferation of standards around the world makes it difficult to know which standard to adopt. City leaders recognize the value and the need to set up data infrastructures that are interoperable and consistent with existing standards, but need guidance and best practices as they move from Charter adoption to implementation of the 6 principles and benefit from greater city-to-city networking opportunities to engage on common urban open data applications and issues. Organized thematically, the newly created open data standards inventory by Geothink and GovEx is designed to help city managers find the standards that they are interested in. The Open Data Institute is also leading a project to facilitate the creation of open standards for data.

Recommendations for Municipalities:

• Publish learning on policy and standard creation for the benefit of other jurisdictions;
• Connect with multi-stakeholder standards initiatives to benefit from their technical and content experience on sector specific standards (i.e. contracting, water, procurement);
• Proactively develop and include sustainable data standardization and maintenance plans into city level open data strategies and operational plans.

Openness in procurement is critical; the way contracts are structured for technology or services can impact the government’s ability to be open by default across units. The issue of procurement was a distinctive theme throughout the interviews, both in the international, and in the Canadian, jurisdictions. Within the broad area of procurement were two common challenges; opening the general government procurement process and opening the procurement of open data and IT products. One interviewee stated,

“Procurement is a very complex and asymmetrical undertaking. It is heavily legislated by all levels of government. Opening the procurement documents is one of the areas with the most room for growth moving forward.” – Elected representative respondent

A refined focus on procurement was repeatedly mentioned as having the most potential for moving towards a working model of open by default that can be replicated and scaled across to other departments. Participants suggested that procurement departments need to eliminate issues of proprietary information related to new contracts, and suggested that all data through requests for proposals be automatically open. Some jurisdictions have made progress in this area, putting ‘open’ principles into their contracts with vendors (e.g. Edmonton and the Government of Canada). These stories should be shared broadly by the open data community.

Open data experts in municipal governments acknowledge that they have significant technology and technical debts that that often hinder their progress. They are dealing with numerous IT systems and a large number of ‘off the shelf’ third party vendor products with proprietary information. While acknowledging the challenge of existing IT, one participant suggested planning for the future. They stressed, “our focus should be future looking and through procurement. IT systems do not reflect the needs of the entire organization”. A common fear expressed throughout the interviews was that if procurement did not change, the open data community would still be struggling with disparate and proprietary systems ten years from now.

“Leaders need to talk to staff about what systems they need and put open by default in the RFP [request for proposal] so there’s no issues around proprietary information related to new contracts. All data through the RFP should be able to be automatically opened.” – Civil society respondent

A political representative acknowledged that change will be difficult, saying, “when you are dealing with procurement to purchase data analysis tools, you are caught up needing to go with the lowest bidder, not necessarily the most agile.” These difficult procurement decisions still require transparency and traceability. There is a need to raise awareness of the Open Contracting Partnership, which is the international multi-stakeholder open data standard focused on procurement with many adopters from leading international cities, as it will improve anti-corruption monitoring and communication of procurement decisions to citizens.

Recommendations for regional and national government:

• Initiate a high-level working group focused on creating agile procurement policies across jurisdictions;
• Collaborate on creating and sharing pre-qualified vendor lists across jurisdictions.

• International Cities: Durham, United States; Buenos Aires, Argentina; Lviv, Ukraine

The Province of Ontario was included in the research because it has an explicit focus on using its adoption of the Charter to support interjurisdictional collaboration with municipalities throughout Ontario.

Lviv, Ukraine, has a city motto of “Open to the World,” and has been ranked by Transparency International-Ukraine as the country’s most transparent city. The city is a new adopter of the Charter. Buenos Aires, Argentina, a member of the Charter since 2015, is part of the Open Government Partnership’s Subnational Pilot Project, and is instrumental in promoting open data throughout the country. The final international city, Durham, United States, has had an open data policy for the past two years and uses the principle of ‘Open by Default’ to guide their aspirations.

There are over ninety municipalities in Canada that release open data. However, this study only engaged with the subset of jurisdictions that have adopted or have engaged in dialogue with the Charter about adoption. This biased selection towards cities interested in international collaboration and the Charter principles.