Compare commits
2 Commits
589547cfed
...
47a01292dd
| Author | SHA1 | Date | |
|---|---|---|---|
| 47a01292dd | |||
| 89d23e0abd |
16
.dockerignore
Normal file
16
.dockerignore
Normal file
@ -0,0 +1,16 @@
|
||||
target/
|
||||
*.class
|
||||
*.jar
|
||||
*.log
|
||||
.git
|
||||
.gitignore
|
||||
.idea
|
||||
.vscode
|
||||
*.iml
|
||||
node_modules/
|
||||
uploads/
|
||||
.env
|
||||
.env.example
|
||||
docker-compose.yml
|
||||
Dockerfile*
|
||||
engines/
|
||||
13
Dockerfile
Normal file
13
Dockerfile
Normal file
@ -0,0 +1,13 @@
|
||||
FROM eclipse-temurin:25-jdk-alpine AS builder
|
||||
WORKDIR /workspace
|
||||
COPY mvnw pom.xml ./
|
||||
COPY .mvn .mvn
|
||||
RUN ./mvnw dependency:go-offline -B
|
||||
COPY src src
|
||||
RUN ./mvnw package -DskipTests -B
|
||||
|
||||
FROM eclipse-temurin:25-jre-alpine
|
||||
WORKDIR /app
|
||||
COPY --from=builder /workspace/target/*.jar app.jar
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["java", "-jar", "app.jar"]
|
||||
292
README.md
Normal file
292
README.md
Normal file
@ -0,0 +1,292 @@
|
||||
# decompile-ai
|
||||
|
||||
**decompile-ai** is a backend service for automated binary analysis. It accepts uploaded binaries, runs them through multiple analysis engines (IDA Pro 5.0, IDA Pro 6.6, Ghidra) and DiE inside Docker containers, then provides AI-powered chat over the decompiled code using Ollama and a RAG pipeline backed by pgvector embeddings.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Component | Technology |
|
||||
|---|---|
|
||||
| Language | **Java 25** |
|
||||
| Framework | **Spring Boot 4.0.6** |
|
||||
| Build | **Maven** (wrapper included) |
|
||||
| Database | **PostgreSQL** with **pgvector** extension |
|
||||
| Migrations | **Flyway** |
|
||||
| Message Broker | **RabbitMQ** |
|
||||
| Module Boundaries | **Spring Modulith** |
|
||||
| AI / Chat | **Spring AI** + **Ollama** |
|
||||
| Container Mgmt | **docker-java** |
|
||||
| Code Formatting | **Spotless** (Google Java Format) |
|
||||
| Test DB | **H2** (in-memory) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ REST API Layer │
|
||||
│ Workspace / Project / Binary / Analysis / Jobs / AI │
|
||||
└─────────────┬───────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────────────────────┐
|
||||
│ Service Layer │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌───────┐ ┌──────────┐ │
|
||||
│ │ Workspace│ │ Analysis │ │ Engine│ │ AI │ │
|
||||
│ │ Service │ │ Service │ │Service│ │ Service │ │
|
||||
│ └──────────┘ └──────────┘ └───────┘ └──────────┘ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌────────────────────┐ │
|
||||
│ │ DiE │ │ Docker │ │ Job (Async Queue) │ │
|
||||
│ │ Service │ │ Service │ │ ┌────────────────┐ │ │
|
||||
│ └──────────┘ └──────────┘ │ │ RabbitMQ Pub/Sub│ │ │
|
||||
│ │ └────────────────┘ │ │
|
||||
└─────────────┬───────────────└────────────────────┘ │
|
||||
│
|
||||
┌─────────────▼───────────────────────────────────────┐
|
||||
│ Infrastructure │
|
||||
│ PostgreSQL + pgvector | RabbitMQ | Ollama │
|
||||
│ Docker Engines: IDA5 | IDA66 | Ghidra | DiE │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Module Boundaries (Spring Modulith)
|
||||
|
||||
Each module declares its allowed dependencies through `package-info.java`:
|
||||
|
||||
- **`workspace`** — CRUD for workspaces, projects, and uploaded binaries. Publishes `BinaryUploadedEvent`.
|
||||
- **`analysis`** — Stores and queries static analysis results (functions, xrefs, labels, segments, strings, structs, enums, imports, libraries). Listens for `StaticAnalysisRequestedEvent`, publishes `StaticAnalysisCompletedEvent`.
|
||||
- **`engine`** — Manages analysis engines (IDA 5, IDA 66, Ghidra) running in Docker containers. Produces standardized `AnalysisResult` output.
|
||||
- **`die`** — Runs DiE (Detect It Easy) inside a Docker container for file type detection and packer identification.
|
||||
- **`docker`** — Low-level Docker container lifecycle management via `docker-java`.
|
||||
- **`job`** — Async job queue: creates, dispatches, executes, and tracks jobs via RabbitMQ. Job types: `ANALYZE_FILE`, `STATIC_ANALYSIS`, `GENERATE_EMBEDDINGS`.
|
||||
- **`ai`** — AI-powered chat (SSE streaming) with RAG over decompiled code. Uses Spring AI + Ollama for chat and embedding generation. Exposes tool functions for the LLM to query analysis data.
|
||||
- **`common`** — Shared configuration, exceptions (`NotFoundException`, `ConflictException`), and global error handling.
|
||||
|
||||
### Job Lifecycle
|
||||
|
||||
```
|
||||
ENQUEUED → STARTED → IN_PROGRESS → COMPLETED
|
||||
↘ → FAILED
|
||||
→ CANCELED
|
||||
```
|
||||
|
||||
### Async Flow (Upload → Analysis → Chat)
|
||||
|
||||
```
|
||||
1. Upload binary → BinaryUploadedEvent
|
||||
2. Job: ANALYZE_FILE → DiE detection + engine analysis (IDA5/IDA66/Ghidra)
|
||||
3. Job: STATIC_ANALYSIS → Parse engine output → store functions/xrefs/labels/etc.
|
||||
4. Job: GENERATE_EMBEDDINGS → Chunk decompiled code → generate pgvector embeddings
|
||||
5. Chat session → RAG retrieval → Ollama streaming response with tool calls
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Java 25**
|
||||
- **PostgreSQL** with **pgvector** extension
|
||||
- **RabbitMQ**
|
||||
- **Ollama** (with models pulled: e.g. `gemma4:12b`, `qwen3-embedding:latest`)
|
||||
- **Docker** (with access to `/var/run/docker.sock` from within the app container)
|
||||
|
||||
## Getting Started
|
||||
|
||||
### 1. Clone and Configure
|
||||
|
||||
```bash
|
||||
git clone <repo-url>
|
||||
cd decompile-ai/backend
|
||||
cp .env.example .env # edit as needed
|
||||
```
|
||||
|
||||
### 2. Start Infrastructure
|
||||
|
||||
```bash
|
||||
docker compose up -d postgres rabbitmq
|
||||
```
|
||||
|
||||
This starts PostgreSQL (with pgvector) and RabbitMQ. Healthchecks ensure they're ready before the backend starts.
|
||||
|
||||
### 3. Create the Database
|
||||
|
||||
```sql
|
||||
CREATE DATABASE decompile_ai;
|
||||
CREATE EXTENSION vector;
|
||||
```
|
||||
|
||||
Or let Flyway handle schema creation on first run (the `vector` extension must be installed manually or via superuser).
|
||||
|
||||
### 4. Pull Ollama Models
|
||||
|
||||
```bash
|
||||
ollama pull gemma4:12b
|
||||
ollama pull qwen3-embedding:latest
|
||||
```
|
||||
|
||||
### 5. Run the Application
|
||||
|
||||
```bash
|
||||
# Development
|
||||
./mvnw spring-boot:run
|
||||
|
||||
# Or with Docker Compose (full stack)
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
The API will be available at `http://localhost:8080`.
|
||||
|
||||
Swagger UI is available at `http://localhost:8080/swagger-ui.html`.
|
||||
|
||||
## Build Commands
|
||||
|
||||
```bash
|
||||
./mvnw compile # Compile
|
||||
./mvnw test # Run tests
|
||||
./mvnw test -Dtest=ChatServiceTest # Run a specific test class
|
||||
./mvnw package -DskipTests # Package as JAR
|
||||
./mvnw spotless:apply # Format code (Google Java Format)
|
||||
./mvnw spotless:check # Check formatting (CI)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration is in `src/main/resources/application.yml` and `.env`.
|
||||
|
||||
### Key Settings
|
||||
|
||||
| Property | Default | Description |
|
||||
|---|---|---|
|
||||
| `app.storage.upload-dir` | `./uploads/binaries` | Uploaded binary storage |
|
||||
| `app.docker.enabled` | `true` | Enable Docker container management |
|
||||
| `app.die.image` | `decompile-ai/diec:latest` | DiE Docker image |
|
||||
| `app.die.timeout-seconds` | `60` | DiE analysis timeout |
|
||||
| `app.engines.ida5.image` | `decompile-ai/ida5:latest` | IDA Pro 5.0 image |
|
||||
| `app.engines.ida5.timeout-seconds` | `300` | IDA 5 analysis timeout |
|
||||
| `app.engines.ida66.image` | `decompile-ai/ida66:latest` | IDA Pro 6.6 image |
|
||||
| `app.engines.ida66.timeout-seconds` | `300` | IDA 66 analysis timeout |
|
||||
| `app.engines.ghidra.image` | `decompile-ai/ghidra:latest` | Ghidra image |
|
||||
| `app.engines.ghidra.timeout-seconds` | `600` | Ghidra analysis timeout |
|
||||
| `app.ai.default-chat-provider` | `ollama` | Chat provider (`ollama`) |
|
||||
| `app.ai.embedding.dimension` | `4096` | Embedding vector dimension |
|
||||
| `app.ai.embedding.chunk-max-chars` | `3000` | Max chars per embedding chunk |
|
||||
| `app.ai.embedding.batch-size` | `20` | Embedding generation batch size |
|
||||
| `app.ai.rag.top-k` | `5` | RAG top-K results |
|
||||
| `app.ai.rag.similarity-threshold` | `0.6` | Minimum similarity score |
|
||||
| `app.ai.rag.max-context-tokens` | `8000` | Max context tokens for RAG |
|
||||
| `app.ai.chat.max-history-messages` | `20` | Max chat history messages |
|
||||
| `spring.servlet.multipart.max-file-size` | `500MB` | Max upload size |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `AI_DEFAULT_CHAT_PROVIDER` | `ollama` | Chat provider |
|
||||
| `AI_CHAT_MODEL_OLLAMA` | `gemma4:12b` | Ollama chat model |
|
||||
| `AI_EMBEDDING_MODEL` | `qwen3-embedding:latest` | Ollama embedding model |
|
||||
| `AI_EMBEDDING_DIMENSION` | `4096` | Embedding dimension |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
A complete, interactive API reference is available via Swagger UI when the application is running:
|
||||
|
||||
```
|
||||
http://localhost:8080/swagger-ui.html
|
||||
```
|
||||
|
||||
### Module Overview
|
||||
|
||||
| Module | Base Path | Description |
|
||||
|---|---|---|
|
||||
| Workspaces | `/api/workspaces` | Workspace CRUD (top-level grouping) |
|
||||
| Projects | `/api/workspaces/{id}/projects` | Project CRUD (nested under workspaces) |
|
||||
| Binaries | `/api/projects/{id}/binaries` | Binary upload, download, and management |
|
||||
| Analysis | `/api/binaries/{id}/analysis` | Static analysis results (functions, xrefs, labels, strings, structs, enums, imports, libraries, segments) |
|
||||
| Jobs | `/api/jobs` | Async job queue — list, filter, get status, cancel |
|
||||
| AI Chat | `/api/chat/sessions` | Chat sessions and SSE streaming messages |
|
||||
| Engines | `/api/engines` | Available analysis engines |
|
||||
|
||||
## AI Chat Features
|
||||
|
||||
- **Streaming responses** via Server-Sent Events (SSE)
|
||||
- **Tool calling** — the LLM can query functions, xrefs, labels, strings, structs, imports, and more in real time
|
||||
- **Mutation suggestions** — the AI can propose renames, type changes, and comments; mutations can be applied via the API
|
||||
- **RAG pipeline** — decompiled code is chunked, embedded via Ollama, stored in pgvector, and retrieved at query time for context-aware responses
|
||||
- **Chat history** — full conversation history is persisted and included in context (up to `max-history-messages`)
|
||||
|
||||
### Stream Event Types
|
||||
|
||||
The SSE stream emits a sealed hierarchy of events:
|
||||
- `chunk` — text delta from the LLM
|
||||
- `thinking` — reasoning tokens
|
||||
- `tool_call_start` / `tool_call_end` — tool invocation boundaries
|
||||
- `mutation_suggested` / `mutation_applied` — rename/type/comment changes
|
||||
- `title_generated` — auto-generated session title
|
||||
- `done` — stream complete
|
||||
- `error` — error details
|
||||
|
||||
## Docker Images
|
||||
|
||||
The project includes Dockerfiles for the analysis engines:
|
||||
|
||||
| Image | Base | Engine |
|
||||
|---|---|---|
|
||||
| `Dockerfile` | `eclipse-temurin:25-jre-alpine` | Main app (multi-stage) |
|
||||
| `Dockerfile.die` | `ubuntu:24.04` | DiE v3.21 |
|
||||
| `engines/ida5/Dockerfile.ida5` | `debian:bullseye-slim` | IDA Pro 5.0 + Wine 32-bit |
|
||||
| `engines/ida66/Dockerfile.ida66` | `debian:bookworm-slim` | IDA Pro 6.6 + Wine 32-bit + Python 2.7 |
|
||||
| `engines/ghidra/Dockerfile` | `alpine:latest` | Ghidra (placeholder) |
|
||||
|
||||
Build engine images before running the full stack:
|
||||
|
||||
```bash
|
||||
docker build -t decompile-ai/diec:latest -f Dockerfile.die .
|
||||
docker build -t decompile-ai/ida5:latest engines/ida5/
|
||||
docker build -t decompile-ai/ida66:latest engines/ida66/
|
||||
docker build -t decompile-ai/ghidra:latest engines/ghidra/
|
||||
```
|
||||
|
||||
## Database Migrations
|
||||
|
||||
Managed by Flyway in `src/main/resources/db/migration/`. Migrations are numbered sequentially (`V001__...` through `V024__...`) and cover all schema objects including pgvector indexes for embedding similarity search.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
./mvnw test # All tests
|
||||
./mvnw test -Dtest=*IntegrationTest # Integration tests only
|
||||
./mvnw test -Dtest=*Test # Unit tests only
|
||||
```
|
||||
|
||||
Tests use:
|
||||
- **JUnit 5** + **Mockito** for unit tests (`@ExtendWith(MockitoExtension.class)`)
|
||||
- **Spring Boot Test** + **H2** + **MockMvc** for integration tests (`@SpringBootTest`, `@ActiveProfiles("test")`)
|
||||
- **Spring Modulith Test** for architecture verification (`ModulithArchitectureTest`)
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/main/java/ai/decompile/
|
||||
├── DecompileAiApplication.java # Entry point
|
||||
├── WebConfig.java # CORS configuration
|
||||
├── common/ # Shared utilities, exceptions, error handling
|
||||
├── workspace/ # Workspace → Project → Binary CRUD
|
||||
├── analysis/ # Static analysis results (functions, xrefs, labels, etc.)
|
||||
├── engine/ # Analysis engines (IDA5, IDA66, Ghidra)
|
||||
├── die/ # DiE file type detection
|
||||
├── docker/ # Docker container management
|
||||
├── job/ # Async job queue (RabbitMQ)
|
||||
└── ai/ # AI chat + RAG embeddings (Spring AI + Ollama)
|
||||
└── tools/ # LLM tool functions for querying analysis data
|
||||
```
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
See [AGENTS.md](AGENTS.md) for the full style guide. Key highlights:
|
||||
|
||||
- **Null checks**: `Objects.isNull()` / `Objects.nonNull()` — never raw `== null`
|
||||
- **Braces**: Always required, even for single-line `if` bodies
|
||||
- **Type inference**: `var` for local variables when type is obvious from RHS
|
||||
- **DI**: Constructor injection via Lombok `@RequiredArgsConstructor`
|
||||
- **Transactions**: `@Transactional` on writes, `@Transactional(readOnly = true)` on reads
|
||||
- **Logging**: Lombok `@Log4j2` with parameterized messages
|
||||
- **Formatting**: Google Java Format via Spotless (`./mvnw spotless:apply`)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@ -1,4 +1,24 @@
|
||||
services:
|
||||
backend:
|
||||
container_name: decompile-ai-backend
|
||||
build: .
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/decompile_ai
|
||||
- SPRING_RABBITMQ_HOST=rabbitmq
|
||||
- APP_STORAGE_UPLOAD_DIR=/tmp/decompile-ai/uploads/binaries
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- /tmp/decompile-ai/uploads:/tmp/decompile-ai/uploads
|
||||
networks:
|
||||
- decompile-ai-network
|
||||
|
||||
postgres:
|
||||
container_name: decompile-ai-postgres
|
||||
image: pgvector/pgvector:pg18
|
||||
@ -10,15 +30,25 @@ services:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres:/var/lib/postgresql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U decompile_ai"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- decompile-ai-network
|
||||
|
||||
rabbit-mq:
|
||||
rabbitmq:
|
||||
container_name: decompile-ai-rabbitmq
|
||||
image: rabbitmq:4-management
|
||||
ports:
|
||||
- "5672:5672"
|
||||
- "15672:15672"
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- decompile-ai-network
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user