maxun爬虫机器人介绍与部署

软件介绍

机器人爬虫工具，绕开编码，直接从网页中截图并且进行解析

一款全新的无代码网页数据提取平台，无需编程即可轻松抓取网站的数据，支持列表/文本抓取、截图、自定义代理、自动处理分页和滚动等功能。作为一个新的开源项目，它的功能还在不停迭代，计划推比如适应网站布局变化和登录后数据提取等新功能

代码地址：

https://github.com/getmaxun/maxun?tab=readme-ov-file

软件部署

1、下载git

yum install git

2、克隆代码

git clone https://github.com/getmaxun/maxun.git

3、创建配置文件

cd maxun mkdir .env

添加如下内容

# App Setup NODE_ENV=production # Set to 'development' or 'production' as required JWT_SECRET=a9Z$kLq7^f03GzNw!bP9dH4xV6sT2yXl3O8vR@uYq3 # Replace with a secure JWT secret key DB_NAME=maxun # Your PostgreSQL database name DB_USER=postgres # PostgreSQL username DB_PASSWORD=postgres # PostgreSQL password DB_HOST=postgres # Host for PostgreSQL in Docker DB_PORT=5432 # Port for PostgreSQL (default: 5432) ENCRYPTION_KEY=f4d5e6a7b8c9d0e1f23456789abcdef01234567890abcdef123456789abcdef0 # Key for encrypting sensitive data (passwords and proxies) MINIO_ENDPOINT=minio # MinIO endpoint in Docker MINIO_PORT=9000 # Port for MinIO (default: 9000) MINIO_CONSOLE_PORT=9001 # Web UI Port for MinIO (default: 9001) MINIO_ACCESS_KEY=minio_access_key # MinIO access key MINIO_SECRET_KEY=minio_secret_key # MinIO secret key REDIS_HOST=redis # Redis host in Docker REDIS_PORT=6379 # Redis port (default: 6379) # Backend and Frontend URLs and Ports BACKEND_PORT=8080 # Port to run backend on. Needed for Docker setup FRONTEND_PORT=5173 # Port to run frontend on. Needed for Docker setup BACKEND_URL=http://localhost:8080 # URL on which the backend runs. You can change it based on your needs. PUBLIC_URL=http://localhost:5173 # URL on which the frontend runs. You can change it based on your needs. VITE_BACKEND_URL=http://localhost:8080 # URL used by frontend to connect to backend. It should always have the same value as BACKEND_URL VITE_PUBLIC_URL=http://localhost:5173 # URL used by backend to connect to frontend. It should always have the same value as PUBLIC_URL # Optional Google OAuth settings for Google Sheet Integration GOOGLE_CLIENT_ID=your_google_client_id GOOGLE_CLIENT_SECRET=your_google_client_secret GOOGLE_REDIRECT_URI=your_google_redirect_uri # Telemetry Settings - Please keep it enabled. Keeping it enabled helps us understand how the product is used and assess the impact of any new changes. MAXUN_TELEMETRY=true

方式一：

Docker部署minio-CSDN博客

Docker部署Redis教程-CSDN博客

Docker部署Postgres教程_docker pull postgres-CSDN博客

Linux部署NodeJS-CSDN博客

项目部署

git clone https://github.com/getmaxun/maxun # change directory to the project root cd maxun # install dependencies npm install # change directory to maxun-core to install dependencies cd maxun-core npm install # get back to the root directory cd .. # make sure playwright is properly initialized npx playwright install npx playwright install-deps # get back to the root directory cd .. # start frontend and backend together npm run start

方式二：

a、下载docker compose

yum -y update yum install -y docker-compose-plugin yum install -y python-pip docker compose version

b、启动容器

docker compose --env-file .env up -d

4、测试

浏览器中输入http://localhost:5173/

如果远程，需要讲localhost改成服务器IP

部署参考：

https://github.com/getmaxun/maxun