Federal bodies are obtaining multiple proprietary AI formulas for duties that may influence individuals’ physical security and civil liberties without having access to comprehensive insights on the operation or training of these systems, according to recently disclosed data.

Customs and Border Protection and the Transportation Security Administration lack records concerning the quality of the data deployed to create and assess algorithms that examine travelers’ bodies for risks, as per the agencies’ 2024 AI inventory documents.

The Veterans Health Administration is in the midst of obtaining an algorithm from a private entity intended to foresee chronic diseases among veterans, but the agency mentioned it is “unclear how the company procured the data” on veterans’ medical documents it utilized to train the model.

And for over 100 algorithms that may influence people’s safety and rights, the agency employing the models didn’t have entry to source code indicating how they function.

As the upcoming Trump administration plans to eliminate newly implemented regulations for federal AI procurement and security, the inventory data reveal the extent to which the government has come to lean on private firms for its most high-risk AI systems.

“I’m profoundly concerned about proprietary systems that wrest influence away from agencies to administer and provide benefits and services to the public,” expressed Varoon Mathur, who until recently was a senior AI consultant to the White House overseeing the AI inventorying process. “We must collaborate hand in hand with proprietary vendors. Often that’s advantageous, yet often we’re unsure of their activities. And if we lack control over our data, how shall we handle risk?”

Internal analyses and external examinations have identified significant faults with some federal agencies’ high-risk algorithms, including a racially biased model the IRS employed to decide which taxpayers to audit and a VA suicide prevention algorithm that favored white men over other demographics.

The 2024 inventories offer the most comprehensive examination thus far of the federal government’s usage of artificial intelligence and its knowledge concerning those systems. For the first time since the inventorying commenced in 2022, agencies had to respond to several inquiries about whether they possessed model documentation or source code access and whether they had evaluated the risks tied to their AI systems.

Out of the 1,757 AI systems reported by agencies for utilization over the year, 227 were identified as likely to affect civil liberties or physical security, and more than half of those high-risk systems were entirely crafted by commercial providers. (For 60 of these high-risk systems, agencies didn’t supply details on their creators. Some agencies, including the Department of Justice, Department of Education, and Department of Transportation haven’t published their AI inventories yet, and military and intelligence bodies are exempted from this requirement).

For no less than 25 systems impacting safety or rights, agencies reported a lack of documentation concerning maintenance, structure, quality, or intended usage of the training and evaluation data. For at least 105, agencies indicated they didn’t have source code access. Agencies didn’t address the documentation inquiry for 51 tools or the source code query for 60 tools. Some high-risk systems remain in the developmental or procurement stages.

Under the Biden administration, the Office of Management and Budget (OMB) introduced fresh guidelines obligating agencies to conduct comprehensive evaluations of risky AI systems and to guarantee that contracts with AI vendors permit access to essential information regarding the models, which may include training data documentation or the code itself.

These directives are more robust than anything AI sellers are likely to face when marketing their products to other enterprises or to state and regional governments (although numerous states will deliberate on AI safety legislations in 2025) and government software vendors have resisted these, asserting that agencies should decide what kind of assessment and transparency is necessary individually.

“Trust but verify,” expressed Paul Lekas, chief of global public policy for the Software & Information Industry Association. “We’re cautious about imposing burdensome requirements on AI developers. Simultaneously, we acknowledge the necessity for some scrutiny regarding what degree of transparency is required to cultivate the trust necessary for government utilization of these tools.”

The U.S. Chamber of Commerce, in remarks provided to the OMB concerning the new guidelines, stated “the government shouldn’t demand any specific training data or data collections on AI models that the government procures from vendors.” Palantir, a prominent AI supplier, suggested that the federal government should “steer clear of overly directing rigid documentation instruments, and instead allow AI service providers and vendors the freedom to define context-specific risk.”

Instead of access to training data or source code, AI vendors argue that in most situations, agencies should be satisfied with model scorecards—documents that characterize the data and machine learning methods an AI model utilizes but don’t include technical specifics that businesses deem trade secrets.

Cari Miller, who has assisted in formulating international benchmarks for acquiring algorithms and co-established the nonprofit AI Procurement Lab, portrayed the scorecards as a lobbyist’s proposal that is “not a poor starting point, but merely a starting point” for what suppliers of high-risk algorithms ought to be contractually obliged to disclose.

“Procurement is one oflas mecanismos más significativos de gobernanza, es donde realmente se pone a prueba, es la entrada principal, es donde puedes determinar si dejas entrar o no lo malo,” comentó ella. “Necesitas comprender si los datos en ese modelo son representativos, si son sesgados o no. ¿Qué hicieron con esos datos y de dónde provienen? ¿Acaso todos provienen de Reddit o Quora? Porque si es así, puede que no sean lo que necesitas.”

Como señaló la OMB al presentar sus reglas sobre IA, el gobierno federal es el mayor comprador individual en la economía de EE. UU., encargado de más de $100 mil millones en adquisiciones de IT en 2023. La dirección que tome respecto a la IA — lo que solicita a los proveedores que revelen y cómo examina los productos antes de adoptarlos — probablemente establecerá el estándar sobre cuán transparentes deben ser las empresas de IA en relación a sus productos al vender a agencias gubernamentales más pequeñas o incluso a otras empresas privadas.

El presidente electo Trump ha manifestado claramente su plan de anular las reglas de la OMB. Hizo campaña con una plataforma del partido que pedía “derogar [la] peligrosa Orden Ejecutiva de Joe Biden que obstaculiza la Innovación en IA e impone ideas radicales de izquierda en el desarrollo de esta tecnología.”

Mathur, el exasesor principal de la Casa Blanca sobre IA, expresó su deseo de que la administración entrante no cumpla con esa promesa y destacó que Trump inició esfuerzos para construir confianza en los sistemas de IA federales con su orden ejecutiva en 2020.

Lograr que las agencias inventarien sus sistemas de IA y respondan preguntas sobre los sistemas propietarios que utilizan fue una tarea monumental, señaló Mathur, que ha sido “profundamente útil” pero que requiere continuidad.

“Si no tenemos el código o los datos o el algoritmo, no vamos a poder entender el impacto que estamos teniendo,” indicó.