Bloop

BLOOP AI has developed innovative tools to modernize legacy COBOL codebases, enabling developers to maintain and scale critical mainframe applications using modern programming languages and development practices. Through their comprehensive transformation pipeline and advanced AI-powered development environment, BLOOP helps organizations bridge the gap between legacy systems and modern software engineering methodologies while maintaining the functional integrity of their code. This technical exploration examines BLOOP's key modernization tools, AI benchmark framework, and data security practices that enable secure, efficient COBOL application development in today's evolving technological landscape.

Terms of Service

Account Requirements and Security

Users must provide complete and accurate information for their account, including email address verification. BLOOP reserves the right to contact users about product details and marketing communications, which users can opt-out of. The company emphasizes user responsibility for maintaining account security, including proper password management. BLOOP explicitly states that it is not liable for damages resulting from users' acts or omissions.

Age and Contract Requirements

The Services are not targeted at children and have specific age requirements, with users needing to be at least 13 years old (or 16 in Europe) to use the Services. All users must demonstrate legal capacity to enter contracts. The company maintains that all users are at least 16 years old unless otherwise verified.

Content Responsibility

BLOOP AI LIMITED has established content policies regarding user-generated content, including text, photos, videos, audio, code, computer software, and items for sale. The company maintains that it has not reviewed all user content on their platforms and is not responsible for content use or third-party website links. This policy applies to all content posted to or through their services.

AI-Powered Legacy Code Conversion

Bloop's modernization solution consists of three primary components: COBOLEval, bloop modernize, and the codebase understanding tool.

COBOLEval: AI Benchmark for COBOL

The company developed COBOLEval as a benchmark for large language models (LLMs) to assess their performance on COBOL tasks. This Python-based benchmark converts 146 challenging problems from the HumanEval suite into COBOL, using six test cases per problem. The evaluation harness employs the open-source GnuCOBOL compiler to test model outputs.

bloop modernize: AI Conversion Pipeline

bloop modernize automates the COBOL to Java conversion process while maintaining functional equivalence. The company's approach involves analyzing the original codebase's behavior as ground truth and validating each refactor through automated tests. The conversion pipeline supports target-platform agnosticism, enabling the production of readable Java code for z/OS, cloud environments, or other platforms.

Codebase Understanding Tool

The updated codebase understanding tool provides enhanced support for mainframe languages, including COBOL. This improvement enables natural language question-asking and code navigation capabilities specifically designed for mainframe projects. The toolset helps bridge the gap between legacy COBOL codebases and modern development practices.

The company's technical foundation rests on decades of experience in machine learning model training and compiler development. Their approach combines static transpilation with AI-driven refactoring to produce maintainable Java code that preserves the original COBOL program's functionality. Each method undergoes localized refactoring to reduce line count before entire classes are rewritten within the 4096 token limit.

COBOL Development Environment

Bloop's modernization tools provide developers with a comprehensive environment for secure and efficient COBOL application development. The company's approach combines advanced AI technologies with decades of compiler development expertise to bridge the gap between legacy COBOL codebases and modern development practices.

Code Modernization Pipeline

At the core of Bloop's development environment is their AI-driven modernization pipeline, which consists of three key components: COBOLEval, bloop modernize, and the codebase understanding tool. These tools work together to transform COBOL codebases into maintainable Java applications while preserving functional equivalence.

Natural Language Development Tools

The updated codebase understanding tool enables developers to work more efficiently with mainframe languages through improved natural language processing capabilities. This enhanced toolset supports direct code navigation and question-asking functionality specifically designed for mainframe projects. These features help bridge the knowledge gap between legacy COBOL code and modern development methodologies.

AI Copilot Integration

As part of their development environment, Bloop provides a coding copilot powered by their own AI model, mAInframer-1. This offline copilot tool helps developers maintain productivity while working with legacy COBOL codebases by offering intelligent assistance and suggestions during the coding process.

Modernization Workflow

The modernization process begins with an initial survey that risk-scores each component and dependency in the codebase. Following this evaluation, Bloop implements a proof-of-concept conversion of a portion of the system to demonstrate immediate value. Once these preliminary steps are complete, the full modernization process can begin, with teams onboarding to work with the modernized codebase and focus on innovation rather than maintenance tasks.

AI Evaluation Benchmarks

The benchmarking framework introduced by Bloop, known as COBOLEval, represents a significant advance in measuring Large Language Model (LLM) performance specifically for COBOL tasks. This evaluation platform converts 146 challenging problems from the HumanEval suite into COBOL format, with each problem accompanied by an average of six test cases.

The COBOL-specific benchmark structure presents unique programming language challenges. Unlike modern languages that employ local variables, COBOL requires all variables to be declared ahead of time in the WORKING-STORAGE SECTION. This static declaration requirement influences the evaluation process, particularly when dealing with variable out-of-order situations. To address this complexity, COBOLEval employs two primary evaluation approaches. The first uses a simplified approach where the WORKING-STORAGE SECTION and PROCEDURE DIVISION are generated in sequence and then reinserted into the correct positions. The second approach leverages program decomposition into prefix, middle, and suffix sections, allowing for more flexible code generation that can fill in gaps within existing program structures.

Evaluation metrics focus on both successful compilation and functional correctness. Using the open-source GnuCOBOL compiler, the evaluation harness verifies that generated solutions execute correctly and pass specified test cases. The benchmark results demonstrate substantial performance improvements after model fine-tuning. While GPT-4 achieves pass@1 scores of 47.94% and compile rates of 73.97%, Bloop's mAInframer series models significantly outperform, with passes reaching 69.17% and compile rates of 69.17% for the 7b variant and 73.97% for the 34b variant.

The evaluation process also highlights the limitations of current AI technology when applied to complex legacy languages. Although models show significant improvement over their open-source counterparts, they still face challenges with more intricate COBOL tasks. This is evidenced by the gap between AI performance (10.27% of problems solved correctly) and human performance (67% success rate), as well as the relatively high failure rate of GPT-4 generated solutions when compiled with GnuCOBOL (47.94%).

Data Privacy and Security

Bloop AI Limited acts as the data controller for all personal data processed through their website or services. This includes collection, storage, and usage of data such as names, email addresses, phone numbers, IP addresses, and web usage patterns. The company maintains appropriate security measures to protect this information and allows users to opt out of account-related communications.

Data Collection and Usage

Users can expect their personal data to be collected and used for account management, service delivery, and product improvement purposes. The company follows standard industry practices for data protection, implementing technical and organizational measures to secure personal information against unauthorized access or disclosure. All data retention periods adhere to legal requirements and internal policies designed to minimize data storage duration.

User Rights and Compliance

Bloop's privacy framework complies with relevant data protection regulations, including the General Data Protection Regulation (GDPR) for European users. Users retain fundamental rights under data protection law, including the ability to access their personal data, request corrections, or ask for their information to be deleted. The company provides clear instructions for exercising these rights through their contact channels, with all communications handled securely and in accordance with legal standards.

Security Practices

The company employs standard security protocols to protect personal data, including regular security audits and data breach response procedures. In the unlikely event of a security incident, Bloop maintains processes for notifying affected users and implementing remediation steps promptly. The company's privacy policy undergoes regular reviews to ensure compliance with evolving legal standards and technological best practices.