Microservice & MDM Strategies For Agile Architecture
The InfoSec Consulting Series #31
By Jay Pope
How can Master Data Management (MDM) and microservices coexist? Agile thinking tells us that microservices must be developed by autonomous teams, with the authority to choose the most appropriate data store. Meanwhile, MDM practitioners and tools are trying to reconcile multiple copies of data, to unravel discrepancies. As architects and developers, we can see the benefits and challenges of both technologies. Especially if we are repurposing a large unwieldy enterprise application.
Here we consider strategies for implementing microservices and securing microservices workloads, both for repurposing a monolith and for new development. We also consider how the microservices strategy touches on MDM and whether the two approaches are really at odds.
Repurposing – From Monolith To Microservice
To the company auditors, enterprise-level applications are an asset, a tangible piece of intellectual property on the balance sheet. To middle management they are a puzzle, constantly demanding development resource. Change requests result in developers sucking in air between their teeth. As architects, we understand both perceptions; we can see their business value, but also the cost and complexity of maintaining and extending them.
The benefits of cloud technology, coupled with the need for enhanced security have forced many enterprises to re-examine their application architectures. Rather than persist with the increasing complexity, some teams have taken the plunge to repurpose to a new architecture. One that supports the use of modern technologies such as cloud, but which also enables the team to work in a more agile manner. To break the monolith into microservices.
Breaking the monolith fits perfectly with Agile development. Each piece can be developed and maintained by a small team and multiple teams can work concurrently without interference. Discrete functions can be developed in short development sprints, without recourse to overall architecture consultations. Breaking the monolith requires the adoption of best practices in 3 key areas, data architecture, application architecture and packaging/deployment.
Use a data-centric approach to application-wide objects such as Purchase Order. Build a core group of APIs to provide access to these objects. This will provide consistent access and behaviour regardless of the technology used to access them. This is a classic MDM approach, as we will see later. Break up the database into smaller groups of related tables. This will give some autonomy to the Agile team along with opening the options for choice of technology. It may be possible to move those tables into a technology that is more appropriate to the data relationships.
REST/JMS services will need to be disentangled from their Web Application Archive (WAR). Begin by deploying each service using its own WAR. The service can then be refactored as necessary without affecting its peers. SOAP/EJB services should be re-implemented as a RESTful interface. This requires refactoring them into an asset-based service and using JSON to define their object representations. Servlet/JSP interfaces should be rewritten to build a domain layer that can be represented as a RESTful service. The Servlet can then be refactored to refer to the new service or replaced with new code.
SOA services will need to be re-implemented by defining the business logic as RESTful services. They should be decomposed to adopt a strategy of having one container per service. It’s vital that all new services are catalogued, using a tool such as Swagger. Services will need to be analysed from a security standpoint. Network traffic between the data entre and the internet will be secured, but communications within the data centre between applications and servers (E-W Security) also need to be considered.
Adopt the principle of independent WARs, rather than bundling related WARs into an Enterprise Archive (EAR). This brings flexibility to the teams deploying the system and allows more options on the use of physical servers. Each WAR can be managed separately, perhaps using an automated pipeline. Follow the strategy of one container per service by deploying each WAR in its own server and if possible in its own Docker container. This builds in flexibility, as the containers can be scaled separately in future. Use an orchestration technology such as Kubernetes to manage the server infrastructure and for securing microservices workloads.
MDM Strategies For Agile Teams
Microservices can bring flexibility and autonomy to an Agile team. They can be developed quickly and changes can be made without breaking other areas of code. However, microservices can bring their own challenges to the development process, notably around data redundancy. As each microservice handles its own persistence and multiple microservices handle intersecting pieces of data, there is likely to be an issue with data redundancy. This is bound to have an impact on MDM.
Traditionally, the purpose of MDM is to provide a business with a single point of reference for data. A data analyst can use an MDM tool such as Talend to inspect data in a database, file or cloud application. This can help to build a picture as to where attention is needed. Subsequently, processes may be developed to clean and distribute data throughout the business. Data will be actively managed at entity level to provide a “single source of the truth”. It’s a central tenet of Agile and microservices that each service has its own data store. In fact, the team should be able to choose the database or data store that best suits their service. This not only optimises performance, but ensures that teams do not fall over one another when making schema changes. However, there is a side effect that databases can become out-of-sync and foreign key lookups can fail.
It’s important then, that an MDM monitor is used during development, running alongside the continuous build, to identify and possibly correct inconsistencies. A 3rd party tool may be suitable, but it will need to have polyglot access across all the technologies. Alternatively, a tool may be developed in-house, to run alongside the unit tests and other validation processes.
Microservices vs MDM?
While it seems that microservices and MDM are at odds, it’s important to remember that the data and its structure are not static. The business needs its data to be governed and organised, but the data must be available to staff when they need it. As the microservices-based architecture takes shape, progressively more and more data will be accessed through these services, assisting MDM and making the physical storage location irrelevant. Further, as the business grows and changes, the data will be able to evolve with it.
Does Your Organisation Need Top Cyber Security Consultants?
We are a team of experts with extensive knowledge and experience of helping organisations improve business performance. Our highly qualified consultancy team can deliver cyber security capability at all levels of your organisation and are on hand to help ensure your projects deliver solutions that are appropriately aligned to your cyber security risk position, and meet technical, business and ethics due diligence requirements. Schedule a call above to learn more about how we can help.