Enterprise AI Analysis

To investigate citizens' perceptions, a mixed-method Wizard-of-Oz study (n=43) was conducted using a simulated Intelligent Self-Service Kiosk (ISSK). Participants completed two tasks with varying decision stakes: a low-stakes ID card renewal (regulatory offense, small fine) and a high-stakes social housing application (requesting additional living space, significant personal impact).

For each task, participants experienced one of three decision-making configurations:

• AI Alone: The ISSK independently made and communicated the decision.
• Supervisory Hybrid: AI made the decision, which was then reviewed and confirmed (or rejected) by a civil servant.
• Advisory Hybrid: AI recommended a decision, while a civil servant reviewed the case and made the final decision.

After each interaction, participants rated fairness (procedural, informational, distributive) and adoption intent, followed by a qualitative interview to gather deeper insights into their experiences and expectations.

Quantitative Analysis: Surprisingly, quantitative analysis found no significant effects of either decision stakes or decision-making configuration on perceived fairness (procedural, informational, distributive) or inclination to adopt. This divergence from prior expectations prompted further qualitative investigation. A key reason for this was identified as participants often misperceiving the actual AI configuration, frequently assuming AI had primary control even in hybrid conditions. This suggests that the *perceived* interaction, not just the stated configuration, drives citizen experience.

Qualitative Analysis: In contrast to the quantitative results, qualitative interviews revealed a rich and nuanced picture, yielding five overarching themes that highlight citizens' expectations and concerns:

1. Interactive Dialogue and Explanation: Citizens felt the ISSK's one-sided dialogue was rigid and inflexible, desiring opportunities to clarify, negotiate, and receive more in-depth, comparative, and counterfactual explanations.
2. Value of Human Touch with Rising Stakes: For low-stakes tasks, AI alone was accepted for efficiency. However, for high-stakes, emotionally charged decisions, human involvement was demanded for empathy, contextual sensitivity, ethical judgment, and negotiation.
3. Human Involvement: Symbolic vs. Functional: Oversight was often perceived as superficial and symbolic, lacking visible and tangible human engagement. Citizens suggested measures like showing the civil servant's name/picture or enabling video calls to make human presence meaningful.
4. Appeals: Where AI Ends and Humans Step In: The right to appeal was crucial, even if unused, especially for high-stakes decisions, seen as a safeguard for empathy and reconsideration. A 'pre-appeal' conversational stage with AI was also desired for clarification.
5. Balancing Efficiency with Human Needs: Citizens acknowledged the trade-off between AI's efficiency and the need for human control, fairness, and the potential for manipulation with low human oversight.

The findings underscore that effective AI deployment in public administration necessitates a citizen-centered approach that prioritizes democratic values alongside efficiency. The quantitative results' non-significance and the rich qualitative data highlight the limits of standard metrics in capturing the nuanced human experience with AI.

1. Foster Interactive Dialogue: Public-sector AI systems must evolve beyond rigid 'submit and wait' workflows. Implement features for iterative clarification, user input confirmation, expansion on personal circumstances, and post-decision dialogue. This makes human-AI interaction more transparent, engaging, and builds trust by ensuring citizens feel heard and understood.

2. Ensure Tangible Human Involvement: For high-stakes or emotionally significant decisions, human oversight must be visible and meaningful, not merely symbolic. Design systems that support active collaboration and empathetic engagement between AI and civil servants. This could include explicit handovers, optional video calls, or even avatars signaling human presence, reinforcing trust in human expertise.

3. Human-Led Appeals: The right to contest AI-based decisions is a crucial safeguard. Appeals processes should remain human-led to ensure legitimacy and empathy. AI can support a 'pre-appeal' stage, offering explanations and clarifications, but the final appeal mechanism should involve human caseworkers who can exercise discretion and provide personalized attention.

4. Balance Efficiency with Democratic Values: Resist the tendency to sacrifice discretion for pure efficiency. AI systems should augment civil servants' judgment, enhancing empathy, contestability, and citizens' rights. Implement AI incrementally, starting with low-stakes tasks, to build citizen familiarity and confidence before extending to more complex, discretionary domains.