When asking a board for results by the end of the quarter for some AI pilot, one should also investigate what went wrong for most of the others. Most failed AI pilots were not failed by the model. Rather, the data was spread across six systems, the integration layer failed under the load, or there was no way to identify who had approved the training data. All the technical groundwork that every use case is built on top of was not done.
Below are the key infrastructure, security, data, and integration issues that must be resolved prior to committing budget to deliver any specific use case. Further, the issue of ageing systems that impede organisations will be addressed.
Sizing infrastructure against the workload you actually have
Inference workloads are often very different from typical business workloads (e.g. Transactional processing). They can be very bursty and latency sensitive and very expensive to run badly. Therefore it’s very important to determine in advance of provisioning any infrastructure whether you are planning to call a hosted API to a model, fine tune a model on your own data or run inference locally on a device (for latency reasons or data residency reasons). Each of these different use cases has very different infrastructure requirements.
Be aware of the high cost of network egress (the outgoing traffic from within your network to an external service) for large document sets, image libraries etc. Even if you initially modelled the cost of the AI model running on your documents, the cost of transferring those documents to the model could end up far exceeding the cost of the model itself. Measure this cost first.
Questions worth answering before provisioning
- What is the acceptable response time for the end user, and does that rule out a round trip to a public cloud region?
- Where does the data need to physically reside for contractual or regulatory reasons?
- Can the workload tolerate spot or reserved capacity, or does it need on-demand throughout?
- What happens to cost if usage grows tenfold, a realistic outcome for a tool people like?
Security controls that assume the model will be probed
AI systems create a whole new surface area to attack. So in addition to regular penetration testing, you should test for prompt injection attacks, training data poisoning attacks and model output manifold attacks that leak sensitive information. Test your AI assistant for these types of attacks before you deploy it to your customers.
Ensure that appropriate Identity and Access Management controls are in place for the AI agent accessing the document store. In retrieval scenarios, appropriate permissions for the user requesting access to the information must be enforced at retrieval time, not indexing time. Thus, rapidly growing volumes of corporate documents (e.g. Emails) can be prevented from being surfaced in response to appropriate questions from anyone, including innocuous AI agents inappropriately granted broad read access to all corporate documents by earlier, amateur IT deployments.
Logging and retention
Ensure prompt history, retrieved documents and generated content are all logged in the same way as other financial transactions and be able to reconstruct what happened in error cases. Define the retention period for prompt logs, they are a sensitive data store.
Getting data into a state a model can use
Data readiness is often the area in which timelines slip, as much of the work is detailed, time-consuming and not particularly ‘glamorous’ to work on. Set realistic scopes for each phase of work and capture the true complexity of any tasks.
| Readiness stage | What it looks like | Typical effort before AI use |
| Fragmented | Data in departmental silos, no shared definitions, heavy duplication | Six to twelve months of consolidation |
| Consolidated | Central warehouse or lake, inconsistent quality and ownership | Three to six months on quality and stewardship |
| Governed | Defined owners, lineage tracked, classification applied | Weeks to prepare use-case-specific sets |
| AI-ready | Labelled, access-controlled, pipelines monitored for drift | Ready for production deployment |
Note especially that for classification into public, internal, confidential, regulated and similar categories the corresponding labelled data is needed to enforce adequate boundaries when retrieving or training data with AI systems.
Integration and the systems holding you back
AI delivers most value when you embed it in your workflows. Therefore a big part of the required groundwork sits with your integration layer. The finance platform API should still be there, the application should still be patched by the vendor and authentication should still work with your company ID provider. If any of these criteria aren’t met, it will hold back your AI initiatives.
One way to approach auditing your application portfolio is to define three simple criteria, namely does it have a documented API surface, is the application still being security patched by the vendor, and can the authentication be brought under your central identity provider? If an application fails more than two of these simple test cases then it likely needs to be deprecated and replaced by a new application or wrapped by middleware.
Where managed IT services fit
Sustained modernisation is typically outside of the capabilities of running organisations and their internal teams. This is why, in addition to data engineering and the design of specific use cases, it is good to outsource work such as patching, monitoring, managing identities and managing the life-cycles of the underlying IT infrastructure to a managed IT services provider. This enables you to retain your internal specialists on the work where they truly add value.
Sequencing the work
- Inventory data sources, owners and classification gaps.
- Close identity and access weaknesses in systems the AI will read from.
- Prove one integration path end to end on a low-risk use case.
- Model running costs at realistic adoption volumes, then decide on scale.
Each of the steps identified here needs to be addressed one at a time and each needs to be completed before moving on to the next one. If a step is not addressed then the problem will simply move further down the line and be even more difficult to fix in the long run.
