Enterprise AI Analysis
“It's Just a Wild, Wild West”: Harnessing Public Procurement as an Al Governance Mechanism
Public sector AI has the potential to harm citizens, with risks increasing as its use expands. Recent work positions public procurement as a way to shape public sector AI in line with public interests, using the state's purchasing power to influence which AI systems are procured and under what conditions. This paper examines how this potential can be realised in practice by drawing on semi-structured interviews with UK and EU buyers, providers, and procurement experts. Our findings result in six promising procurement practices that enable the public sector to shape AI in line with public interests, alongside concrete mechanisms to support their uptake. Further, we find that AI-specific procurement approaches remain immature and systems often enter through informal channels with less scrutiny. We provide directions for both research and practice on how public procurement can be used as a governance mechanism for better aligning AI with public interests.
Key Takeaways for Your Enterprise
Understand the critical insights from this research and their implications for your organization's AI strategy.
0
Expert Interviews Conducted
0
Promising Practices Identified
0
EU Digital Decade Investment
Public procurement for AI is largely underutilized and lacks AI-specific approaches. AI systems often enter public sector via informal channels (function creep, pilots) with less scrutiny. A lack of clear guidance and organizational capacity are key barriers to effective AI procurement. Strong vendor influence and skewed competition favor large tech suppliers. Six promising procurement practices identified to align AI with public interests. HCI scholarship has a crucial role in advancing AI governance through procurement practices.
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
AI Procurement Landscape
Key Challenges
Promising Practices
HCI Opportunities
The current state of AI procurement in the public sector is characterized by immaturity and fragmentation. Most AI systems are acquired indirectly, often folded into standard IT procurement processes or through existing framework contracts, which lack AI-specific safeguards. This leads to reduced scrutiny and favors large, established suppliers. Additionally, AI enters the public sector through less transparent channels like updates to existing systems (function creep), embedded AI in connected products ('hidden' AI), and pilot projects which often operate under reduced oversight and can extend indefinitely. This 'wild west' scenario makes it difficult to track AI technologies and ensure public interest alignment.
Four core challenges prevent effective AI procurement aligned with public interests:
- Lack of clear guidance: No consistent vision or actionable guidance on how to procure AI, leading to uncertainty and inconsistent policies. Traditional IT procurement methods are ill-suited for AI's dynamic nature.
- Limited organizational capacity & AI literacy: Public buyers lack skills to evaluate AI systems, identify risks, or scrutinize vendor claims, leading to over-reliance on tech companies and 'inferiority complex'. Procurement teams are often overworked and disconnected from use cases.
- Strong vendor influence & skewed competition: Private vendors, especially large tech firms, actively shape narratives and steer public bodies towards their off-the-shelf solutions, often exploiting the AI 'hype'. Structural barriers like complex tender requirements and framework agreements disadvantage SMEs.
- Complex infrastructures & vendor lock-in: AI systems are embedded within broader IT environments, leading to issues with data ownership, IP governance, and high costs/risks for switching providers. Lack of planning for decommissioning further entrenches dependencies.
Six promising practices can help align public sector AI with public interests:
- Provide clarity on vision, guidance, and operationalisation: Establish a central, actionable vision for AI use, with practical, adaptable guidance and AI-specific contractual clauses.
- Harness insights through knowledge sharing & collaboration: Create hubs, libraries, and collaborative platforms for sharing experiences, best practices, and even failures across government levels.
- Prepare people and culture for careful AI procurement: Build interdisciplinary teams, upskill procurement staff with AI expertise, and foster a culture of openness and critical assessment.
- Focus on defining outcome/problem rather than prescribing solutions: Engage in stakeholder dialogues early on, organize SME-inclusive market dialogues, and use innovative procurement processes for iterative experimentation.
- Consider system interactions and reuse: Strengthen data and IP governance, plan for early exit, ensure modularity and interoperability, and reuse existing (open-source) systems where appropriate.
- Implement quality control measures: Integrate rAI aspects into tender evaluations, conduct meaningful impact assessments (initial and ongoing), and use flexible contracting mechanisms for post-contract monitoring.
HCI scholarship has a critical role in advancing AI governance through procurement by:
- Contributing to the design of soft law instruments and public sector guidance.
- Facilitating systematic deliberations in public sector AI procurement, especially involving citizens, through stakeholder involvement and participatory design.
- Developing effective ways to upskill public sector buyers in AI literacy.
- Applying socio-technical audit scholarship to assess AI systems alignment with public interests and monitor past deployment.
- Co-designing tools and methods for responsible AI development adapted for public sector procurement.
- Balancing iterative development/experimentation/pilots with contractual rigidity and stringent assessments, cultivating openness to failure.
- Leveraging user needs and preferences to shape procured AI systems towards public interest.
Public Procurement Process Overview
Preparation & Pre-Procurement
→
Tendering & Evaluation
→
Contract Awarded
→
Post-Contract Award
→
End of Contract
| Term | Description |
|---|---|
| Buyer | The public sector body that makes the purchase (formally: the contracting authority), e.g., a government department, local authority, hospital, or school. |
| Supplier / Vendor | A private-sector company or organisation that offers to provide the requested works, goods, or services. |
| Contract notice / call for tenders | The public (and published) announcement of a planned purchase that describes what will be procured and how bids will be evaluated. |
| Tender / RFP (Request for Proposals) | The supplier's formal bid submitted in response to the buyer's requirements; the RFP is the buyer's document inviting and specifying those bids. |
| Tendering process | The formal procedure through which suppliers submit bids and the buyer evaluates them against predefined criteria. |
| Framework agreement / contract | An umbrella agreement with pre-selected suppliers that sets terms for future orders over a fixed period. |
| Pilot / proof-of-concept (POC) | A time-limited trial to test feasibility/fit before scale. |
The 'Wild West' of Public Sector AI Acquisition
Our research reveals that formal, AI-specific procurement is nascent. AI systems often enter the public sector via less scrutinized channels, leading to a 'wild west' scenario with unclear safeguards and fragmented practices. This includes:
- Function Creep: AI functionalities incrementally introduced through updates to existing digital services, often unnoticed and without explicit procurement.
- Hidden AI: AI embedded covertly in connected products (e.g., vehicles, traffic lights) without being declared as such, obstructing oversight.
- Pilot Projects: Frequently used to bypass rigorous procurement procedures by staying below financial thresholds, often extending indefinitely without proper evaluation or clear exit strategies.
This approach favors large, established suppliers and contributes to a lack of transparency and accountability in public sector AI adoption.
| ID | Role | Focus Region | Sector | Level | Experience |
|---|---|---|---|---|---|
| P1 | Public-sector procurement adviser at a leading global consulting firm | UK | Consulting | All levels | 7-10 years |
| P2 | Co-leader of a research institute focused on socially beneficial AI | Belgium | Research institute | National, all levels | Not reported |
| P3 | Public-sector procurement consultant with experience on both government and vendor sides | Germany | Private sector | National, all levels | 4-6 years |
| P4 | Coordinator of a European municipal network on innovative procurement | Netherlands | Public sector | Local government + mediator | 7-10 years |
| P5 | Global digital transformation leader at a major international engineering and advisory firm | UK | Private sector + ex-government | All levels | >10 years |
| P6 | Supports local authorities with digital improvement through a national local-government association | UK | NGO / local gov. association | Local government | 4-6 years |
| P7 | Policy and data specialist within a national ministry for international development | Germany | Government agency | National government | Unknown |
| P8 | Lawyer specialising in technology, data governance, and digital policy | Netherlands | Local & national government | All levels | >10 years |
| P9 | Adviser on public-sector innovation and former UK government digital service official | UK, International | Research / consulting + ex-public sector | National government + multilateral | >10 years |
| P10 | Researcher on societal impacts of data and AI at a prominent public-interest research institute | UK | Research institute | National, local government | 1-3 years |
| P11 | Academic working on ethical AI for cities | UK, Netherlands | Academia + research | Local government | 1-3 years |
| P12 | Digital innovation lead within a city administration | Italy | Public sector | Local government | 4-6 years |
| P13 | Supports public-interest organisations with digital governance | EU | Intergovernmental org | EU-level, local government | 4-6 years |
| P14 | Former senior national government official responsible for public administration | Slovenia | Private sector + ex-government | National, all levels | >10 years |
| P15 | Policy coordinator for AI and algorithmic governance within a national interior ministry | Netherlands | Public sector | National, local government | 1-3 years |
| P16 | Legal researcher focusing on complex contracting environments for the public sector | UK | Research institute | All levels | 4-6 years |
| Practice | Implementation Mechanism | Procurement Phase |
|---|---|---|
| Provide clarity on vision, guidance, and operationalisation | Central and actionable vision defining AI and desired AI use | Cross-cutting |
| Provide clarity on vision, guidance, and operationalisation | Guidance that clarifies how to operationalise such a vision | Cross-cutting |
| Harness insights through knowledge sharing & collaboration in the public sector | Hubs / libraries to bundle existing support and positive & negative learnings | Cross-cutting |
| Harness insights through knowledge sharing & collaboration in the public sector | Collaborative platforms and networks for pooling expertise and demand | Cross-cutting |
| Harness insights through knowledge sharing & collaboration in the public sector | Joint procurements (via buyer groups) | Preparation & pre-procurement |
| Prepare people and culture for careful AI procurement | Building interdisciplinary teams | Cross-cutting |
| Prepare people and culture for careful AI procurement | Upskilling procurement-related teams (to enable them to scrutinise offers) | Preparation & pre-procurement |
| Focus on defining the outcome/problem rather than prescribing solutions | Stakeholder dialogues from very early procurement stages onwards to define the desired outcome | Preparation & pre-procurement |
| Focus on defining the outcome/problem rather than prescribing solutions | Organising SME-inclusive pre-procurement market dialogues | Preparation & pre-procurement |
| Focus on defining the outcome/problem rather than prescribing solutions | Innovative procurement processes allowing iterative experimentation | Preparation & pre-procurement |
| Consider system interactions and reuse | Strengthening data and IP governance | Tendering & evaluation |
| Consider system interactions and reuse | Reuse of existing (open-source) systems | Preparation & pre-procurement |
| Consider system interactions and reuse | Creating system repositories | Cross-cutting, end of contract |
| Consider system interactions and reuse | Mapping existing technology stack | Preparation & pre-procurement |
| Implement quality control measures during the entire procurement lifecycle including responsible AI requirements | Integrate quality control measures into the tender | Tendering & evaluation |
| Implement quality control measures during the entire procurement lifecycle including responsible AI requirements | Meaningful impact assessments beyond technical focus | Tendering & evaluation, post-contract award |
| Implement quality control measures during the entire procurement lifecycle including responsible AI requirements | Flexible contracting mechanisms | Tendering & evaluation, post-contract award |
| Implement quality control measures during the entire procurement lifecycle including responsible AI requirements | Facilitate interoperability, openness, modularity, and the ability to switch or exit | Tendering & evaluation |
| Research Question | Relevance |
|---|---|
| How can HCI contribute to the design of soft law instruments and public sector guidance that shape the regulatory scene around AI procurements? | AI Governance, Policy Design |
| How can HCI scholarship around stakeholder involvement and participation facilitate systematic deliberations in public sector AI (pre-)procurement – especially for the involvement of citizens? | Participatory Design, Public Values |
| What are effective ways to upskill public sector buyers in AI literacy, drawing on HCI scholarship? | AI Literacy, Capacity Building |
| How can the public sector apply HCI scholarship on socio-technical audits to assess whether AI systems align with public interests, including monitoring past deployment? | Socio-technical Audits, Accountability |
| How can co-designed tools or methods for responsible AI development be applied or adapted to support public sector AI procurement? | Responsible AI, Tooling |
| How should upskilling procurement teams in AI-specific knowledge, interdisciplinary teams, and consulting external AI experts be balanced? | Interdisciplinary Collaboration, Expertise |
| How can regulations and guidance around public procurement profit from insights into collaborative policy design? | Policy Design, Bottom-up Governance |
| How can iterative development/experimentation/pilots be balanced with contractual rigidity and stringent assessments, drawing from HCI design research? Importantly, how can openness to failure be cultivated thereby? | Agile Procurement, Innovation |
| How do the needs and preferences of different public sector stakeholders influence what is procured or automated and how can this be leveraged towards the public interest? | Stakeholder Needs, Public Interest |
| Research Question | Goal |
|---|---|
| What types of tools can increase the leverage of smaller government agencies towards larger providers? (e.g. platforms for supplier reviews, knowledge sharing, or collaborative tendering) | Empower Smaller Agencies |
| How can shared AI procurements be realised in practice? (e.g. multi-stakeholder dialogues, collaborative procurement models) | Enable Collaborative Procurement |
| How can we ensure that we learn from reflections about completed public sector AI procurements, leveraging collective, distributed intelligence? | Foster Continuous Learning |
| How can SMEs be empowered to contribute to a fair and open market? | Promote Fair Competition |
Calculate Your Potential AI ROI
Estimate the efficiency gains and cost savings your enterprise could achieve by strategically implementing AI.
Estimated Annual Savings
$0
Hours Reclaimed Annually
0
Your AI Implementation Roadmap
A strategic, phased approach to integrating AI governance and procurement best practices within your organization.
Phase 1: Vision & Guidance Alignment
Establish a clear, actionable AI vision for your public sector entity. Integrate AI-specific procurement guidelines, drawing from successful frameworks and legal requirements. Foster cross-departmental alignment on AI goals and ethical considerations.
Phase 2: Capacity Building & Collaboration
Upskill procurement teams and decision-makers in AI literacy and responsible AI practices. Build interdisciplinary teams connecting technical experts, legal, and procurement. Implement knowledge-sharing platforms to leverage collective intelligence across public bodies.
Phase 3: Outcome-Focused Procurement Design
Shift from solution-prescriptive tenders to outcome-focused requirements. Engage stakeholders, including citizens, early in the process to define desired outcomes. Design innovative procurement processes that allow for iterative experimentation and SME inclusion.
Phase 4: System Integration & Lifecycle Management
Prioritize data and IP governance, ensuring clear ownership and ethical use. Design for modularity, interoperability, and early exit strategies to prevent vendor lock-in. Explore and promote reuse of existing open-source or proven AI systems.
Phase 5: Continuous Quality Control & Monitoring
Integrate robust quality control measures and responsible AI requirements into all tender evaluations. Implement meaningful impact assessments, both pre- and post-contract. Establish flexible contracting mechanisms for ongoing monitoring, adaptation, and accountability.
Ready to Transform Your AI Procurement?
Our experts can help you navigate the complexities of AI procurement, ensuring your systems align with public interests and deliver maximum value.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat