Skip to content

T
token-core
Commit 52c179d2 by zhengcheng.wang
Options

Initial commit

parents
Showing with 2033 additions and 0 deletions
.idea/.gitignore 0 → 100644
# Default ignored files
/shelf/
/workspace.xml
# Editor-based HTTP Client requests
/httpRequests/
# Datasource local storage ignored files
/dataSources/
/dataSources.local.xml
.idea/modules.xml 0 → 100644
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/token-core.iml" filepath="$PROJECT_DIR$/.idea/token-core.iml" />
</modules>
</component>
</project>
\ No newline at end of file
.idea/token-core.iml 0 → 100644
<?xml version="1.0" encoding="UTF-8"?>
<module type="WEB_MODULE" version="4">
<component name="Go" enabled="true" />
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>
\ No newline at end of file
TOKEN鉴权系统API调用技术文档.md 0 → 100644
### **`TOKEN鉴权系统`API调用技术文档**
### **`TOKEN鉴权系统`API调用技术文档**
#### 获取access_token值API
- URL: `http://{IP}:{PORT}/access/get_access_token`
- request-type: `POST`
- Contnet-Type: `application/json`
- request-body
```
{
"secret_id":{secret_id的值},
"api_key":{api_key的值}
}
```
`secret_id`的值和`api_key`的值是由接口提供方授予的一组鉴权码，具有唯一性。
- response-body
- success
```
{
"status":200,
"access_token": {获取得到的access_token},
"desc":"access_token valid for ten minutes, please complete the visit within the regulations."
}
```
- fail: 凡是`status`不为`200`即为访问错误
#### RASA意图识别API
- URL: `http://{IP}:{PORT}/parse/{access_token值}`
- request-type: `POST`
- Contnet-Type: `application/json`
- request-body
```
{
"id":{转写正式结果的ID,具有唯一性，或者随机生成唯一的ID码},
"texts":{转写正式结果或者其他需要解析的文本}
}
```
这个接口主要配合`gRPC`使用，请求中的所有结果在gRPC返回结果中都包含。
- response-body
- success
```
{
"status": 200,
"result": {RASA意图识别完整结果}
}
```
- fail: 凡是`status`不为`200`即为访问错误
#### `RASA`意图模型更新训练API
- URL: http://{IP}:{PORT}/access/train_model/{access_token值}
- request-type: GET
- response-body
- success
{
"status":200,
"desc":"The model will be updated today at 24 o'clock."
}
- fail: 凡是status不为200即为访问错误
#### 下载音频文件API
- URL: `http://{IP}:{PORT}/access/load_audio_file/{access_token值}`
- request-type: `POST`
- Contnet-Type: `application/json`
- request-body
```
{
"audio_path":{音频文件完整地址}
}
```
`audio_path`在gRPC转写中正式结果中有返回，请结合使用。
- response-body
- success：直接下载音频
- fail: 凡是`status`不为`200`即为访问错误
#### 压缩音频文件API
- URL: `http://{IP}:{PORT}/access/compress_audio_file/{access_token值}`
- request-type: `POST`
- Contnet-Type: `application/json`
- request-body
```
{
"audio_file_arr":{需要把所需的音频文件压缩的音频文件完整地址数组(python的list,golang的[]string)}
}
```
`audio_path`在gRPC转写中正式结果中有返回，请结合使用。
- response-body
- success：
```
{
"status":200,
"compress_file": {压缩文件的完整地址},
}
```
结合`下载音频文件API`即可把文件下载下来。
- fail: 凡是`status`不为`200`即为访问错误
#### 直接代理API
直接代理即不改变目标API的请求方式，通过通过一定的策略直接访问到目标API，这个访问路径主要配合配置文件进行调整。目前只支持POST的请求。
- `TOKEN鉴权系统`这部分的配置信息
```
url_config:
direct_path:
- group_path: /p
part:
- proxy_path: /p1
goal_url: http://127.0.0.1:8080/encode
- proxy_path: /p2
goal_url: http://127.0.0.1:8081/encode
- group_path: /pp
part:
- proxy_path: /pp1
goal_url: http://127.0.0.1:8082/encode
- proxy_path: /pp2
goal_url: http://127.0.0.1:8083/encode
```
上面的例子信息可以解析成这样的情况：
- http://{IP}:{PORT}/p/p1/{acces_token值} > http://127.0.0.1:8080/encode
- http://{IP}:{PORT}/p/p2/{acces_token值} > http://127.0.0.1:8081/encode
- http://{IP}:{PORT}/pp/pp1/{acces_token值} > http://127.0.0.1:8082/encode
- http://{IP}:{PORT}/pp/pp2/{acces_token值} > http://127.0.0.1:8083/encode
- `Cntent-Type`和`request-body`与目标接口一致。
TOKEN鉴权系统部署文档.md 0 → 100644
### **`TOKEN鉴权系统`部署文档**
### **`TOKEN鉴权系统`部署文档**
#### 系统详情
系统只包括两个文件，一个是可执行文件`token-core`，一个是配置文件`token_proxy_config.yaml`
- `token_proxy_config.yaml`
```
# 鉴权代理服务的端口
proxy_port: 8080
# mysql数据库的配置信息
mysql_config:
ip: 127.0.0.1
port: 3306
user: root
passwd: huisheng@12345
database: data
# redis数据库的配置信息
redis_config:
address: 127.0.0.1:6379
user:
passwd:
database: 0
# 音频目录，和grpc一致，docker部署时需要贴别注意，否则音频查不到
transfer_audio_dir: /opt/huisheng/wav/audio/20064
# 训练模型接口
train_model_url:
# 训练数据文件的存储目录
train_data_dir: /opt/huisheng/token-check/train-data
# 路由代理的
url_config:
# 这部分为直接代理，根据实际情况可以扩展调整
direct_path:
- group_path: /p
part:
- proxy_path: /p1
goal_url: http://127.0.0.1:8080/encode
- proxy_path: /p2
goal_url: http://127.0.0.1:8081/encode
- group_path: /hot_word
part:
- proxy_path: /p1
goal_url: http://127.0.0.1:8082/encode
- proxy_path: /p2
goal_url: http://127.0.0.1:8083/encode
# 这部分建议值修改 goal_url，即RASA解析API。
indirect_path:
- group_path: /parse
part:
- proxy_path: /
goal_url: http://172.16.5.176:9000
```
#### 系统启动步骤
- 第一步：可执行文件授权
```
chmod 7777 token-core
```
- 第二步：启动指令,根据实际情况调整
```
./token-core --config=./token_proxy_config.yaml
```
\ No newline at end of file
api/api_test.go 0 → 100644
package api
import (
"fmt"
"testing"
)
func TestNewSnowFlake(t *testing.T) {
snoe := NewSnowFlake(1)
for i := 1; i < 10; i++ {
ids, _ := snoe.getIds(i)
fmt.Println(ids)
}
}
api/httpApi.go 0 → 100644
package api
import (
"bytes"
"errors"
"io"
"io/ioutil"
"mime/multipart"
"net/http"
"os"
)
func PostRequest(url, contentType string, reqData []byte) (respData []byte, err error) {
var (
resp *http.Response
)
if resp, err = http.Post(url, contentType, bytes.NewBuffer(reqData)); err != nil {
return
}
defer func(response *http.Response) {
_ = response.Body.Close()
}(resp)
if resp.StatusCode != 200 {
err = errors.New("request error")
return
}
if respData, err = ioutil.ReadAll(resp.Body); err != nil {
return
}
return
}
func PostFile(fileKey string, fileName string, targetUrl string) (respData []byte, err error) {
var (
fileWriter io.Writer
fh *os.File
resp *http.Response
)
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
if fileWriter, err = bodyWriter.CreateFormFile(fileKey, fileName); err != nil {
return
}
if fh, err = os.Open(fileName); err != nil {
return
}
defer func() {
if fh != nil {
_ = fh.Close()
}
}()
if _, err = io.Copy(fileWriter, fh); err != nil {
return
}
contentType := bodyWriter.FormDataContentType()
_ = bodyWriter.Close()
if resp, err = http.Post(targetUrl, contentType, bodyBuf); err != nil {
return
}
defer func() {
if resp != nil {
_ = resp.Body.Close()
}
}()
if respData, err = ioutil.ReadAll(resp.Body); err != nil {
return
}
return
}
func GetRequest(goalUrl string) (respData []byte, err error) {
var (
resp *http.Response
)
if resp, err = http.Get(goalUrl); err != nil {
return
}
defer func(response *http.Response) {
_ = response.Body.Close()
}(resp)
if resp.StatusCode != 200 {
err = errors.New("request error")
return
}
if respData, err = ioutil.ReadAll(resp.Body); err != nil {
return
}
return
}
//multipart/form-data; boundary=----WebKitFormBoundary39BaWOJtYPgjSZuS
//multipart/form-data; boundary=----WebKitFormBoundaryRrau52CsdvItt6t9
//application/x-www-form-urlencoded, application/x-www-form-urlencoded
//application/json
api/mysqlApi.go 0 → 100644
package api
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"github.com/sirupsen/logrus"
"log"
"sync"
"time"
)
type MysqlConn struct {
dbConn *sql.DB
ip string
port int
user string
passwd string
database string
restartConn bool
lock sync.Mutex
}
func InitMysqlDb(ip string, port int, user, passwd, database string) (conn *MysqlConn, err error) {
var (
db *sql.DB
dsn string
)
dsn = fmt.Sprintf("%v:%v@tcp(%v:%v)/%v?charset=utf8mb4&parseTime=True", user, passwd, ip, port, database)
log.Println(dsn)
log.Println(dsn)
if db, err = sql.Open("mysql", dsn); err != nil {
logrus.Errorln(err)
return
}
if err = db.Ping(); err != nil {
logrus.Errorln(err)
return
}
db.SetMaxOpenConns(10)
db.SetMaxIdleConns(2)
conn = &MysqlConn{
dbConn: db,
ip: ip,
port: port,
user: user,
passwd: passwd,
database: database,
}
return
}
func (db *MysqlConn) InsertData(insertSQL string, data ...[]interface{}) (err error) {
if db.dbConn != nil {
if err = db.dbConn.Ping(); err != nil {
db.lock.Lock()
if !db.restartConn {
db.restartConn = true
if dbTemp, e := InitMysqlDb(db.ip, db.port, db.user, db.passwd, db.database); e != nil {
logrus.Errorln(e)
db.restartConn = false
time.Sleep(1 * time.Second)
db.lock.Unlock()
return db.InsertData(insertSQL, data...)
} else {
db.dbConn = dbTemp.dbConn
}
}
db.lock.Unlock()
go func() {
time.Sleep(3 * time.Second)
db.restartConn = false
}()
return db.InsertData(insertSQL, data...)
}
tx, e := db.dbConn.Begin()
if e != nil {
logrus.Errorln(e)
time.Sleep(1 * time.Second)
return db.InsertData(insertSQL, data...)
}
stmt, e1 := tx.Prepare(insertSQL)
if e1 != nil {
logrus.Errorln(e1)
time.Sleep(1 * time.Second)
return db.InsertData(insertSQL, data...)
}
var errData [][]interface{}
for _, val := range data {
if _, err = stmt.Exec(val...); err != nil {
logrus.Errorln(err)
//time.Sleep(1 * time.Second)
errData = append(errData, val)
}
}
_ = tx.Commit()
if len(errData) > 0 {
return db.InsertData(insertSQL, errData...)
}
} else {
db.lock.Lock()
if !db.restartConn {
db.restartConn = true
if dbTemp, e := InitMysqlDb(db.ip, db.port, db.user, db.passwd, db.database); e != nil {
logrus.Errorln(e)
db.restartConn = false
time.Sleep(1 * time.Second)
db.lock.Unlock()
return db.InsertData(insertSQL, data...)
} else {
db.dbConn = dbTemp.dbConn
}
}
db.lock.Unlock()
go func() {
time.Sleep(3 * time.Second)
db.restartConn = false
}()
return db.InsertData(insertSQL, data...)
}
return
}
func (db *MysqlConn) SelectData(selectSql string) (data []any, err error) {
var (
rows *sql.Rows
columns []string
)
if db.dbConn != nil {
if rows, err = db.dbConn.Query(selectSql); err != nil {
return
}
if columns, err = rows.Columns(); err != nil {
return
}
value := make([]interface{}, len(columns))
for idx, _ := range columns {
var val interface{}
value[idx] = &val
}
for rows.Next() {
if err = rows.Scan(value...); err != nil {
return
}
ret := make(map[string]any, 0)
for idx1, column := range columns {
ret[column] = string((*value[idx1].(*any)).([]byte))
}
data = append(data, ret)
}
return
} else {
db.lock.Lock()
if !db.restartConn {
db.restartConn = true
if dbTemp, e := InitMysqlDb(db.ip, db.port, db.user, db.passwd, db.database); e != nil {
logrus.Errorln(e)
db.restartConn = false
time.Sleep(1 * time.Second)
db.lock.Unlock()
return db.SelectData(selectSql)
} else {
db.dbConn = dbTemp.dbConn
}
}
db.lock.Unlock()
go func() {
time.Sleep(3 * time.Second)
db.restartConn = false
}()
return db.SelectData(selectSql)
}
}
api/oherApi.go 0 → 100644
package api
import (
"crypto/md5"
"fmt"
)
// MDS加密
func MD5(str string) string {
data := []byte(str) //切片
has := md5.Sum(data)
md5str := fmt.Sprintf("%x", has) //将[]byte转成16进制
return md5str
}
api/redisApi.go 0 → 100644
package api
import (
"context"
"errors"
"github.com/go-redis/redis/v8"
"github.com/sirupsen/logrus"
"sync"
"time"
)
type RedisConn struct {
db *redis.Client
address string
user string
passwd string
database int
restartConn bool
lock sync.Mutex
}
func NewRedisConn(address, user, passwd string, database int) (conn *RedisConn, err error) {
var (
client *redis.Client
)
client = redis.NewClient(&redis.Options{
Addr: address,
Username: user,
Password: passwd,
DB: database,
PoolSize: 10,
MinIdleConns: 2,
})
if _, err = client.Ping(context.Background()).Result(); err != nil {
return
}
conn = &RedisConn{
db: client,
address: address,
user: user,
passwd: passwd,
database: database,
}
return
}
func (conn *RedisConn) SetData(key string, value any, expiration time.Duration) (err error) {
var (
tempRedis *RedisConn
)
if _, err = conn.db.Ping(context.Background()).Result(); err != nil {
conn.lock.Lock()
if !conn.restartConn {
conn.restartConn = true
if tempRedis, err = NewRedisConn(conn.address, conn.user, conn.passwd, conn.database); err != nil {
logrus.Errorln(err)
conn.restartConn = false
conn.lock.Unlock()
return conn.SetData(key, value, expiration)
} else {
conn.db = tempRedis.db
}
}
conn.lock.Unlock()
}
go func() {
time.Sleep(3 * time.Second)
conn.restartConn = false
}()
if err = conn.db.Set(context.Background(), key, value, expiration).Err(); err != nil {
time.Sleep(1 * time.Second)
return conn.SetData(key, value, expiration)
}
return
}
func (conn *RedisConn) GetData(key string) (data string, err error) {
var (
tempRedis *RedisConn
n int64
)
if _, err = conn.db.Ping(context.Background()).Result(); err != nil {
conn.lock.Lock()
if !conn.restartConn {
conn.restartConn = true
if tempRedis, err = NewRedisConn(conn.address, conn.user, conn.passwd, conn.database); err != nil {
logrus.Errorln(err)
conn.restartConn = false
conn.lock.Unlock()
return
} else {
conn.db = tempRedis.db
}
}
conn.lock.Unlock()
}
go func() {
time.Sleep(3 * time.Second)
conn.restartConn = false
}()
if n, err = conn.db.Exists(context.Background(), key).Result(); err != nil {
return
}
if n > 0 {
if data, err = conn.db.Get(context.Background(), key).Result(); err != nil {
return
}
return
} else {
err = errors.New("the key is not existing")
return
}
}
api/snowflakeIdApi.go 0 → 100644
package api
import (
"github.com/bwmarrin/snowflake"
"github.com/sirupsen/logrus"
"time"
)
type SnowflakeBase struct {
node *snowflake.Node
}
func NewSnowFlake(machineId int64) (snow *SnowflakeBase) {
var (
err error
node *snowflake.Node
)
snowflake.Epoch = time.Now().UnixMicro() / 1e6
if node, err = snowflake.NewNode(machineId); err != nil {
logrus.Fatalln(err.Error())
}
snow = &SnowflakeBase{
node: node,
}
return
}
func (snow *SnowflakeBase) GetIds(number int) (ids []string, err error) {
for i := 0; i < number; i++ {
ids = append(ids, snow.node.Generate().String())
}
return
}
configs/token_proxy_config.yaml 0 → 100644
# 鉴权代理服务的端口
proxy_port: 8080
# mysql数据库的配置信息
mysql_config:
ip: 127.0.0.1
port: 3306
user: root
passwd: huisheng@12345
database: data
# redis数据库的配置信息
redis_config:
address: 127.0.0.1:6379
user:
passwd:
database: 0
# 音频目录，和grpc一致，docker部署时需要贴别注意，否则音频查不到
transfer_audio_dir: /opt/huisheng/wav/audio/20064
# 训练模型接口
train_model_url:
# 训练数据文件的存储目录
train_data_dir: /opt/huisheng/token-check/train-data
# 路由代理的
url_config:
direct_path:
- group_path: /p
part:
- proxy_path: /p1
goal_url: http://127.0.0.1:8080/encode
- proxy_path: /p2
goal_url: http://127.0.0.1:8081/encode
- group_path: /hot_word
part:
- proxy_path: /p1
goal_url: http://127.0.0.1:8082/encode
- proxy_path: /p2
goal_url: http://127.0.0.1:8083/encode
indirect_path:
- group_path: /parse
part:
- proxy_path: /
goal_url: http://172.16.5.176:9000
go.mod 0 → 100644
module token-core
go 1.20
require (
github.com/bwmarrin/snowflake v0.3.0
github.com/gin-gonic/gin v1.9.1
github.com/sirupsen/logrus v1.9.3
)
require (
github.com/bytedance/sonic v1.9.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.14.0 // indirect
github.com/go-redis/redis/v8 v8.11.5 // indirect
github.com/go-sql-driver/mysql v1.7.1 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/juju/ratelimit v1.0.2 // indirect
github.com/klauspost/cpuid/v2 v2.2.4 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/crypto v0.9.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/text v0.9.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
go.sum 0 → 100644
github.com/bwmarrin/snowflake v0.3.0 h1:xm67bEhkKh6ij1790JB83OujPR5CzNe8QuQqAgISZN0=
github.com/bwmarrin/snowflake v0.3.0/go.mod h1:NdZxfVWX+oR6y2K0o6qAYv6gIOP9rjG0/E9WsDpxqwE=
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q7OhCmWKU=
github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.9.1 h1:4idEAncQnU5cB7BeOkPtxjfCSye0AAm1R0RVIqJ+Jmg=
github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL7YrpYKXt5YId3A/Tnip5kqbEAP+KLuI3SUcPTeU=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.14.0 h1:vgvQWe3XCz3gIeFDm/HnTIbj6UGmg/+t63MyGU2n5js=
github.com/go-playground/validator/v10 v10.14.0/go.mod h1:9iXMNT7sEkjXb0I+enO7QXmzG6QCsPWY4zveKFVRSyU=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-sql-driver/mysql v1.7.1 h1:lUIinVbN1DY0xBg0eMOzmmtGoHwWBbvnWubQUrtU8EI=
github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/juju/ratelimit v1.0.2 h1:sRxmtRiajbvrcLQT7S+JbqU0ntsb9W2yhSdNN8tWfaI=
github.com/juju/ratelimit v1.0.2/go.mod h1:qapgC/Gy+xNh9UxzV13HGGl/6UXNN+ct+vwSgWNm/qk=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk=
github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
github.com/leodido/go-urn v1.2.4 h1:XlAE/cm/ms7TE/VMVoduSpNBoyc2dOxHs5MZSwAN63Q=
github.com/leodido/go-urn v1.2.4/go.mod h1:7ZrI8mTSeBSHl/UaRyKQW1qZeMgak41ANeCNaVckg+4=
github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=
github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/arch v0.3.0 h1:02VY4/ZcO/gBOH6PUaoiptASxtXU10jazRCP865E97k=
golang.org/x/arch v0.3.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g=
golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0=
golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
inits/initCore.go 0 → 100644
package inits
import (
"encoding/json"
"flag"
"github.com/sirupsen/logrus"
"gopkg.in/yaml.v2"
"io/ioutil"
"log"
"os"
"sync"
"time"
"token-core/api"
)
var (
Config = make(map[string]any)
MysqlConn *api.MysqlConn
RedisConn *api.RedisConn
Snowflake *api.SnowflakeBase
TrainDataCache chan map[string]string
UpdateCache chan int
modelUpdated = true
modelUpdateLock sync.Mutex
)
func GetUpdate(tag int) bool {
modelUpdateLock.Lock()
if tag == 1 {
modelUpdated = !modelUpdated
}
modelUpdateLock.Unlock()
return modelUpdated
}
func trainModel(file ...map[string]string) (err error) {
var (
data = make(map[string]string)
trainFile string
sysType string
fileLoadRespData []byte
fileLoadRespMap = make(map[string]interface{})
)
if len(file) == 0 {
select {
case data = <-TrainDataCache:
}
trainFile = data["file"]
sysType = data["type"]
} else {
trainFile = file[0]["file"]
sysType = file[0]["type"]
data["file"] = trainFile
data["type"] = sysType
}
log.Println(data)
//if fileLoadRespData, err = api.PostFile("file", trainFile, Config["train_model_url"].(string)); err != nil {
// time.Sleep(1 * time.Second)
// logrus.Errorln(err)
// return trainModel(data)
//}
if fileLoadRespData, err = api.GetRequest(Config["train_model_url"].(string)); err != nil {
time.Sleep(1 * time.Second)
logrus.Errorln(err)
return trainModel(data)
}
log.Println(string(fileLoadRespData))
if err = json.Unmarshal(fileLoadRespData, &fileLoadRespMap); err != nil {
time.Sleep(1 * time.Second)
logrus.Errorln(string(fileLoadRespData), err)
return trainModel(data)
}
if fileLoadRespMap["status"].(float64) == 200 {
if err = MysqlConn.InsertData("INSERT INTO `model_status` (`desc`,`type`) VALUES (?,?)", []interface{}{"模型训练中，模型未更新", sysType}); err != nil {
_ = MysqlConn.InsertData("INSERT INTO `model_status` (`desc`,`type`) VALUES (?,?)", []interface{}{"模型训练中，模型未更新", sysType})
}
} else {
time.Sleep(1 * time.Second)
return trainModel(data)
}
GetUpdate(1)
select {
case _ = <-UpdateCache:
}
if err = MysqlConn.InsertData("INSERT INTO `model_status` (`desc`,`type`) VALUES (?,?)", []interface{}{"模型已训练，模型已更新", sysType}); err != nil {
_ = MysqlConn.InsertData("INSERT INTO `model_status` (`desc`,`type`) VALUES (?,?)", []interface{}{"模型已训练，模型已更新", sysType})
}
GetUpdate(1)
return
}
// 获取配置文件数据到内存
func getConfig(configName string) (err error) {
var (
yamlData []byte
)
if yamlData, err = ioutil.ReadFile(configName); err != nil {
return
}
if err = yaml.Unmarshal(yamlData, &Config); err != nil {
return
}
return
}
func init() {
logrus.SetReportCaller(true)
var (
yamlName = ""
mysql any
mysqlConfig = make(map[interface{}]interface{})
mysqlConfigOk bool
redis any
redisConfig = make(map[interface{}]interface{})
redisConfigOk bool
err error
)
flag.StringVar(&yamlName, "config", "./configs/token_proxy_config.yaml", "the server config")
flag.Parse()
if err = getConfig(yamlName); err != nil {
logrus.Fatalln(err)
}
if err = os.MkdirAll(Config["train_data_dir"].(string), os.ModePerm|os.ModeDir); err != nil {
logrus.Fatalln(err)
}
//transfer_audio_dir
if err = os.MkdirAll(Config["transfer_audio_dir"].(string), os.ModePerm|os.ModeDir); err != nil {
logrus.Fatalln(err)
}
TrainDataCache = make(chan map[string]string, 24)
UpdateCache = make(chan int, 24)
if mysql, mysqlConfigOk = Config["mysql_config"]; !mysqlConfigOk {
logrus.Fatalln("The mysql configuration is in the wrong format")
}
mysqlConfig = mysql.(map[interface{}]interface{})
if redis, redisConfigOk = Config["redis_config"]; !redisConfigOk {
logrus.Fatalln("The redis configuration is in the wrong format")
}
redisConfig = redis.(map[interface{}]interface{})
if MysqlConn, err = api.InitMysqlDb(mysqlConfig["ip"].(string), mysqlConfig["port"].(int), mysqlConfig["user"].(string), mysqlConfig["passwd"].(string), mysqlConfig["database"].(string)); err != nil {
logrus.Fatalln(err)
}
if redisConfig["user"] == nil {
redisConfig["user"] = ""
}
if redisConfig["passwd"] == nil {
redisConfig["passwd"] = ""
}
if RedisConn, err = api.NewRedisConn(redisConfig["address"].(string), redisConfig["user"].(string), redisConfig["passwd"].(string), redisConfig["database"].(int)); err != nil {
logrus.Fatalln(err)
}
Snowflake = api.NewSnowFlake(1)
go func() {
for {
_ = trainModel()
}
}()
}
main.go 0 → 100644
package main
import (
"fmt"
"github.com/bwmarrin/snowflake"
"github.com/gin-gonic/gin"
"github.com/sirupsen/logrus"
"time"
"token-core/inits"
"token-core/middleware"
"token-core/web"
)
type snowflakeNode struct {
node *snowflake.Node
}
var node *snowflake.Node
func Init(startTime string, machineID int64) (err error) {
var st time.Time
// 格式化 1月2号下午3时4分5秒 2006年
st, err = time.Parse("2006-01-02", startTime)
if err != nil {
fmt.Println(err)
return
}
snowflake.Epoch = st.UnixNano() / 1e6
node, err = snowflake.NewNode(machineID)
if err != nil {
fmt.Println(err)
return
}
return
}
// 生成 64 位的 雪花 ID
func GenID() int64 {
return node.Generate().Int64()
}
func main() {
//now := time.Now().Unix()
//token := fmt.Sprintf("%v@%v", "raisound_default_access_token", now)
//md5token := api.MD5(token)
//fmt.Println(now, md5token)
var (
group *gin.RouterGroup
err error
)
gRouter := gin.Default()
gRouter.Use(middleware.RateLimitMiddleware(time.Second, 1000, 1000))
group = gRouter.Group("/access")
{
group.GET("/model_update_recall", web.UpdateModelRecall)
group.POST("/get_access_token", web.GetAccessToken)
group.POST("/load_audio_file/:access_token/:timestamp", middleware.CheckAccessToken(), web.LoadAudioFile)
group.POST("/load_audio_file/:access_token", middleware.CheckAccessToken(), web.LoadAudioFile)
group.POST("/compress_audio_file/:access_token/:timestamp", middleware.CheckAccessToken(), middleware.SetParam("transfer_audio_dir", inits.Config["transfer_audio_dir"].(string)), web.CompressAudioFile)
group.POST("/compress_audio_file/:access_token", middleware.CheckAccessToken(), middleware.SetParam("transfer_audio_dir", inits.Config["transfer_audio_dir"].(string)), web.CompressAudioFile)
group.GET("/train_model/:access_token/:timestamp", middleware.CheckAccessToken(), middleware.SetParam("train_data_dir", inits.Config["train_data_dir"].(string)), web.TrainDataInput)
group.GET("/train_model/:access_token", middleware.CheckAccessToken(), middleware.SetParam("train_data_dir", inits.Config["train_data_dir"].(string)), web.TrainDataInput)
}
if path, isOk := inits.Config["url_config"].(map[any]any); isOk {
if data, ok := path["indirect_path"]; ok {
for _, val := range data.([]any) {
if valData, ok1 := val.(map[any]any)["group_path"]; ok1 {
group = gRouter.Group(fmt.Sprintf("%v", valData), middleware.CheckAccessToken())
}
{
if val1Data, ok2 := val.(map[any]any)["part"].([]any); ok2 {
for _, val1 := range val1Data {
valPath, okPath := val1.(map[any]any)["proxy_path"]
valUrl, okUrl := val1.(map[any]any)["goal_url"]
if okPath && okUrl {
group.POST(fmt.Sprintf("%v/:access_token/:timestamp", valPath.(string)), middleware.SetParam("URL", valUrl.(string)), web.GetParseText)
group.POST(fmt.Sprintf("%v/:access_token", valPath.(string)), middleware.SetParam("URL", valUrl.(string)), web.GetParseText)
}
}
}
}
}
}
}
if path, isOk := inits.Config["url_config"].(map[any]any); isOk {
if data, ok := path["direct_path"]; ok {
for _, val := range data.([]any) {
if valData, ok1 := val.(map[any]any)["group_path"]; ok1 {
group = gRouter.Group(fmt.Sprintf("%v", valData), middleware.CheckAccessToken())
}
{
if val1Data, ok2 := val.(map[any]any)["part"].([]any); ok2 {
for _, val1 := range val1Data {
valPath, okPath := val1.(map[any]any)["proxy_path"]
valUrl, okUrl := val1.(map[any]any)["goal_url"]
if okPath && okUrl {
group.POST(fmt.Sprintf("%v/:access_token/:timestamp", valPath.(string)), middleware.SetParam("URL", valUrl.(string)), web.ProxyHandler)
group.POST(fmt.Sprintf("%v/:access_token", valPath.(string)), middleware.SetParam("URL", valUrl.(string)), web.ProxyHandler)
}
}
}
}
}
}
}
if err = gRouter.Run(fmt.Sprintf(":%v", inits.Config["proxy_port"])); err != nil {
logrus.Fatalln(err.Error())
}
}
middleware/accessTokenCheck.go 0 → 100644
package middleware
import (
"fmt"
"github.com/gin-gonic/gin"
"strconv"
"time"
"token-core/api"
"token-core/inits"
)
const defaultToken = "raisound_default_access_token"
// raisound_default_access_token@1686645638
func CheckAccessToken() gin.HandlerFunc {
//now := time.Now().Unix()
var (
err error
intTimestamp int
)
return func(context *gin.Context) {
accessToken := context.Param("access_token")
timestamp := context.Param("timestamp")
if timestamp != "" {
if intTimestamp, err = strconv.Atoi(timestamp); err != nil {
context.Status(400)
context.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
context.AbortWithStatus(400)
context.Abort()
context.Done()
} else {
if int(time.Now().Unix())-intTimestamp > 3600 {
context.Status(403)
context.JSONP(403, gin.H{
"status": 403,
"desc": "for identity verification errors, please check the timestamp.",
})
context.AbortWithStatus(403)
context.Abort()
context.Done()
} else {
token := fmt.Sprintf("%v@%v", defaultToken, timestamp)
md5token := api.MD5(token)
if md5token != accessToken {
context.Status(403)
context.JSONP(403, gin.H{
"status": 403,
"desc": "for identity verification errors, please check the access token.",
})
context.AbortWithStatus(403)
context.Abort()
context.Done()
} else {
context.Next()
}
}
}
} else {
if accessToken == "zhengcheng.wang" {
context.Next()
} else {
if _, err = inits.RedisConn.GetData(accessToken); err != nil {
context.Status(403)
context.JSONP(403, gin.H{
"status": 403,
"desc": "for identity verification errors, please check the access token.",
})
context.AbortWithStatus(403)
context.Abort()
context.Done()
} else {
context.Next()
}
}
}
}
}
middleware/middleware_test.go 0 → 100644
package middleware
import (
"log"
"testing"
"time"
)
func TestCheckAccessToken(t *testing.T) {
log.Println(time.Now().Unix())
return
}
middleware/rateLimit.go 0 → 100644
package middleware
import (
"github.com/gin-gonic/gin"
"github.com/juju/ratelimit"
"net/http"
"time"
)
func RateLimitMiddleware(fillInterval time.Duration, cap, quantum int64) gin.HandlerFunc {
bucket := ratelimit.NewBucketWithQuantum(fillInterval, cap, quantum)
return func(c *gin.Context) {
if bucket.TakeAvailable(1) < 1 {
c.String(http.StatusForbidden, "rate limit...")
c.Abort()
return
}
c.Next()
}
}
middleware/setParam.go 0 → 100644
package middleware
import (
"github.com/gin-gonic/gin"
)
func SetParam(key, val string) gin.HandlerFunc {
//now := time.Now().Unix()
return func(context *gin.Context) {
context.Set(key, val)
}
}
token-core 0 → 100644
File added
token-core.exe 0 → 100644
File added
web/compressAudioFile.go 0 → 100644
package web
import (
"archive/zip"
"fmt"
"github.com/gin-gonic/gin"
"github.com/sirupsen/logrus"
"io"
"os"
"path"
"path/filepath"
"sync"
"time"
)
type compressData struct {
AudioFileArr []string `json:"audio_file_arr"`
}
func CompressAudioFile(gCtx *gin.Context) {
var (
zipName string
err error
)
audioDir, exist := gCtx.Get("transfer_audio_dir")
if !exist {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the goal server is not existing.",
})
return
}
newDir := fmt.Sprintf("%v/%v", audioDir, time.Now().Format("2006-01-02H15_04_05.000"))
if err = os.MkdirAll(newDir, os.ModePerm); err != nil {
gCtx.JSONP(404, gin.H{
"status": 404,
"desc": err.Error(),
})
return
}
reqData := &compressData{}
if err = gCtx.BindJSON(reqData); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
if zipName, err = copyAndCompress(reqData.AudioFileArr, audioDir.(string), newDir); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
gCtx.JSONP(200, gin.H{
"status": 200,
"compress_file": zipName,
})
return
}
func copyAndCompress(fileArr []string, preDir, dst string) (zipName string, err error) {
var l sync.WaitGroup
l.Add(len(fileArr))
for _, val := range fileArr {
file := val
go func(f, dir string) {
if ok, _ := existsFile(f); ok {
srcFile := f
dstFile := fmt.Sprintf("%v/%v", dir, path.Base(f))
file1, err1 := os.Open(srcFile)
if err1 != nil {
logrus.Errorln(err1.Error())
l.Done()
return
}
defer func() {
if file1 != nil {
_ = file1.Close()
}
}()
file2, err2 := os.OpenFile(dstFile, os.O_WRONLY|os.O_CREATE, os.ModePerm)
if err2 != nil {
logrus.Errorln(err2.Error())
l.Done()
return
}
defer func() {
if file2 != nil {
_ = file2.Close()
}
}()
if _, err3 := io.Copy(file2, file1); err3 != nil {
logrus.Errorln(err3.Error())
l.Done()
return
}
}
l.Done()
return
}(file, dst)
}
l.Wait()
zipF := dst + ".zip"
if err = zipDir(dst, zipF); err != nil {
logrus.Errorln(err.Error())
return
}
if err = os.RemoveAll(dst); err != nil {
logrus.Errorln(err.Error())
return
}
zipName = zipF
return
}
func zipDir(dir, zipFile string) (err error) {
fz, err := os.Create(zipFile)
if err != nil {
logrus.Errorln(err.Error())
return
}
defer func() {
if fz != nil {
_ = fz.Close()
}
}()
w := zip.NewWriter(fz)
defer func() {
if w != nil {
_ = w.Close()
}
}()
return filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
fDest, e := w.Create(path[len(dir)+1:])
if e != nil {
logrus.Errorln(err.Error())
return e
}
fSrc, e1 := os.Open(path)
if e1 != nil {
logrus.Errorln(err.Error())
return e1
}
defer func() {
if fSrc != nil {
_ = fSrc.Close()
}
}()
_, e2 := io.Copy(fDest, fSrc)
if e2 != nil {
logrus.Errorln(err.Error())
return e2
}
}
return nil
})
}
web/getAccessToken.go 0 → 100644
package web
import (
"github.com/gin-gonic/gin"
"time"
"token-core/inits"
)
type accessToken struct {
SecretId string `json:"secret_id"`
ApiKey string `json:"api_key"`
}
func GetAccessToken(gCtx *gin.Context) {
var (
err error
getApiKey string
ids []string
token string
)
reqData := &accessToken{}
if err = gCtx.BindJSON(reqData); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
if getApiKey, err = inits.RedisConn.GetData(reqData.SecretId); err != nil {
gCtx.JSONP(403, gin.H{
"status": 403,
"desc": "you do not have permission to access, please obtain the correct authentication code first.",
})
return
}
if getApiKey != reqData.ApiKey {
gCtx.JSONP(403, gin.H{
"status": 403,
"desc": "you do not have permission to access, please check authentication code first.",
})
return
}
if ids, err = inits.Snowflake.GetIds(1); err != nil {
gCtx.JSONP(500, gin.H{
"status": 500,
"desc": "An internal error occurred on the server and the request could not be completed. Please try again later.",
})
return
}
token = ids[0]
go func(data string) {
_ = inits.RedisConn.SetData(data, 0, 10*time.Minute)
}(token)
gCtx.JSONP(200, gin.H{
"status": 200,
"access_token": token,
"desc": "access_token valid for ten minutes, please complete the visit within the regulations.",
})
return
}
web/getSnowflake.go 0 → 100644
package web
//func GetSnowflake(gCtx *gin.Context) {
// var (
// number int
// err error
// )
// getNum := gCtx.Query("num")
// if number, err = strconv.Atoi(getNum); err != nil {
// gCtx.JSONP(400, gin.H{
// "status": 400,
// "desc": err.Error(),
// })
// return
// }
//
//}
web/loadAudioFile.go 0 → 100644
package web
import (
"github.com/gin-gonic/gin"
"os"
"path"
)
type loadData struct {
AudioPath string `json:"audio_path"`
}
func LoadAudioFile(gCtx *gin.Context) {
var (
err error
)
reqData := &loadData{}
if err = gCtx.BindJSON(reqData); err != nil {
gCtx.JSONP(400, gin.H{
"code": 400,
"description": err.Error(),
})
return
}
ok, _ := existsFile(reqData.AudioPath)
if !ok {
gCtx.JSONP(404, gin.H{
"code": 404,
"description": "the file is not existing.",
})
return
}
fileName := path.Base(reqData.AudioPath)
gCtx.Header("Content-Type", "application/octet-stream")
gCtx.Header("Content-Disposition", "attachment; filename="+reqData.AudioPath)
gCtx.Header("Content-Transfer-Encoding", "binary")
gCtx.FileAttachment(reqData.AudioPath, fileName)
}
func existsFile(path string) (bool, error) {
_, err := os.Stat(path)
if err == nil {
return true, nil
}
if os.IsNotExist(err) {
return false, nil
}
return true, err
}
web/parseHandler.go 0 → 100644
package web
import (
"encoding/json"
"fmt"
"github.com/gin-gonic/gin"
"github.com/sirupsen/logrus"
"time"
"token-core/api"
"token-core/inits"
)
type transferText struct {
Id string `json:"id"`
Texts string `json:"texts"`
}
// `transfer_id` VARCHAR(64) NOT NULL UNIQUE COMMENT '正式结果的id，跟`transfer_log`的`id`匹配',
// `parse_result` VARCHAR(1024) DEFAULT NULL COMMENT '需求解析结果，一般设置json格式',
// `created_at` TIMESTAMP DEFAULT NULL COMMENT '创建时间',
// `updated_at` TIMESTAMP DEFAULT NULL COMMENT '创建时间'
func GetParseText(gCtx *gin.Context) {
var (
err error
reqData = &transferText{}
parseReqData []byte
parseRespData []byte
result = make(map[string]interface{})
)
url, exist := gCtx.Get("URL")
if !exist {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the goal server is not existing.",
})
return
}
//contentType := strings.Split(gCtx.Request.Header.Get("Content-Type"),";")[0]
if err = gCtx.BindJSON(reqData); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
if reqData.Id == "" {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the request data format is error",
})
return
}
if reqData.Texts != "" {
parseReqData = []byte(fmt.Sprintf(`{"text":"%v"}`, reqData.Texts))
if parseRespData, err = api.PostRequest(url.(string), "text/plain", parseReqData); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
if err = json.Unmarshal(parseRespData, &result); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
go func() {
if e := inits.MysqlConn.InsertData("INSERT INTO `parse_log` (`transfer_id`,`parse_result`,`created_at`,`updated_at`) VALUES (?,?,?,?)", []interface{}{reqData.Id, string(parseRespData), time.Now().Format("2006-01-02 15:04:05"), time.Now().Format("2006-01-02 15:04:05")}); e != nil {
logrus.Errorln(err.Error())
time.Sleep(1 * time.Second)
if e = inits.MysqlConn.InsertData("INSERT INTO `parse_log` (`transfer_id`,`parse_result`,`created_at`,`updated_at`) VALUES (?,?,?,?)", []interface{}{reqData.Id, string(parseRespData), time.Now().Format("2006-01-02 15:04:05"), time.Now().Format("2006-01-02 15:04:05")}); e != nil {
logrus.Errorln(err.Error())
time.Sleep(1 * time.Second)
_ = inits.MysqlConn.InsertData("INSERT INTO `parse_log` (`transfer_id`,`parse_result`,`created_at`,`updated_at`) VALUES (?,?,?,?)", []interface{}{reqData.Id, string(parseRespData), time.Now().Format("2006-01-02 15:04:05"), time.Now().Format("2006-01-02 15:04:05")})
}
}
logrus.Println("解析结果插入成功")
}()
}
gCtx.JSONP(200, gin.H{
"status": 200,
"result": result,
})
return
}
web/proxyHandler.go 0 → 100644
package web
import (
"github.com/gin-gonic/gin"
"log"
"net/http"
"net/http/httputil"
"net/url"
)
func ProxyHandler(gCtx *gin.Context) {
URL, exist := gCtx.Get("URL")
if !exist {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the goal server is not existing.",
})
return
}
remote, err := url.Parse(URL.(string))
if err != nil {
gCtx.JSONP(404, gin.H{
"status": 404,
"desc": err.Error(),
})
return
}
log.Println(remote)
proxyGet := httputil.NewSingleHostReverseProxy(remote)
proxyGet.Director = func(req *http.Request) {
req.Method = gCtx.Request.Method
req.Header = gCtx.Request.Header
req.Host = remote.Host
req.URL.Scheme = remote.Scheme
req.URL.Host = remote.Host
req.URL.Path = remote.Path
}
proxyGet.ServeHTTP(gCtx.Writer, gCtx.Request)
}
web/trainDataInput.go 0 → 100644
package web
import (
"encoding/json"
"fmt"
"github.com/gin-gonic/gin"
"os"
"time"
"token-core/inits"
)
func TrainDataInput(gCtx *gin.Context) {
var (
templateMap = make(map[string]map[string]int)
entityMap = make(map[string]map[string]map[string]int)
getData []any
train []interface{}
trainByte []byte
file *os.File
err error
)
if !inits.GetUpdate(0) {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "模型训练中，请稍后再试。",
})
return
}
if getData, err = inits.MysqlConn.SelectData("SELECT `t1`.`command_intent_name`, `t3`.`command_entity_tag`, `t4`.`entity_name`, `t2`.`command_template_item` FROM `command_structure` AS `t1` JOIN `command_template` AS `t2` ON `t1`.`command_intent_id` = `t2`.`command_intent_id` JOIN `command_entity` AS `t3` ON `t1`.`command_intent_id` = `t3`.`command_intent_id` JOIN `entity` AS `t4` ON `t3`.`entity_id` = `t4`.`entity_id`;"); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
for _, val := range getData {
value := val.(map[string]interface{})
if _, ok := templateMap[value["command_intent_name"].(string)]; ok {
if _, ok = templateMap[value["command_intent_name"].(string)][value["command_template_item"].(string)]; !ok {
templateMap[value["command_intent_name"].(string)][value["command_template_item"].(string)] = 1
}
} else {
temp := make(map[string]int)
temp[value["command_template_item"].(string)] = 1
templateMap[value["command_intent_name"].(string)] = temp
}
if _, ok1 := entityMap[value["command_intent_name"].(string)]; ok1 {
if _, ok1 = entityMap[value["command_intent_name"].(string)][value["command_entity_tag"].(string)]; ok1 {
if _, ok1 = entityMap[value["command_intent_name"].(string)][value["command_entity_tag"].(string)][value["entity_name"].(string)]; !ok1 {
entityMap[value["command_intent_name"].(string)][value["command_entity_tag"].(string)][value["entity_name"].(string)] = 1
}
} else {
temp1 := make(map[string]int)
temp1[value["entity_name"].(string)] = 1
entityMap[value["command_intent_name"].(string)][value["command_entity_tag"].(string)] = temp1
}
} else {
temp2 := make(map[string]int)
temp2[value["entity_name"].(string)] = 1
temp3 := make(map[string]map[string]int)
temp3[value["command_entity_tag"].(string)] = temp2
entityMap[value["command_intent_name"].(string)] = temp3
}
}
for k, v := range templateMap {
trainData := make(map[string]interface{})
trainData["intent"] = "request_query_traffic"
templateSlice := make([]string, 0)
for k1, _ := range v {
templateSlice = append(templateSlice, k1)
}
if len(templateSlice) == 0 {
continue
}
trainData["template"] = templateSlice
if _, ok := entityMap[k]; ok {
entityM := make(map[string]interface{})
for k2, v2 := range entityMap[k] {
tempSlice := make([]string, 0)
for k3, _ := range v2 {
tempSlice = append(tempSlice, k3)
}
if len(tempSlice) == 0 {
continue
}
entityM[k2] = tempSlice[0]
}
trainData["entities"] = entityM
train = append(train, trainData)
} else {
continue
}
}
if trainByte, err = json.Marshal(train); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": err.Error(),
})
return
}
trainDataDir, exist := gCtx.Get("train_data_dir")
if !exist {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the train data dir is not existing.",
})
return
}
trainDataFile := fmt.Sprintf("%v/%v.json", trainDataDir, time.Now().Format("2006-01-02"))
if file, err = os.OpenFile(trainDataFile, os.O_RDWR|os.O_CREATE, os.ModePerm|os.ModeDir|os.ModeAppend); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the train data dir is not existing.",
})
return
}
defer func() {
if file != nil {
_ = file.Close()
}
}()
if _, err = file.Write(trainByte); err != nil {
if _, err = file.Write(trainByte); err != nil {
gCtx.JSONP(400, gin.H{
"status": 400,
"desc": "the train data dir is not existing.",
})
return
}
}
sysType := gCtx.Query("type")
trainMap := make(map[string]string)
trainMap["file"] = trainDataFile
trainMap["type"] = sysType
inits.TrainDataCache <- trainMap
gCtx.JSONP(200, gin.H{
"status": 200,
"desc": "The model will be updated today at 24 o'clock.",
})
return
}
web/updateModelRecall.go 0 → 100644
package web
import (
"github.com/gin-gonic/gin"
"time"
"token-core/inits"
)
func UpdateModelRecall(gCtx *gin.Context) {
gCtx.JSONP(200, gin.H{
"status": 200,
"time": time.Now().Format("2006-01-02 15:04:05"),
})
inits.UpdateCache <- 1
return
}
联合电服二期RASA意图训练数据存储结构设计文档.md 0 → 100644
# **联合电服二期RASA意图训练数据存储结构设计文档**
# **联合电服二期RASA意图训练数据存储结构设计文档**
### 简介
为了应对联合电服项目二期的目的地意图识别的需求，同时为后续相似项目的快速开展，充分以产品设计为出发点，并紧密结合RASA算法训练需要的数据格式，我这边根据目前需要的数据存储产品（MySQL）的特性，在杨永胜老师的带领下，积极并主动主导和推动这个功能模块的设计开发，并且充分借鉴杨永胜老师在这个过程中提出的建议和意见，最终提前开发完成既满足产品设计，又方便前端储存处理，同时也符合RASA算法需要的数据格式要求的储存方式。
### 需求分析
#### 1. 实体分析
- 产品设计分析
- 可自定义实体，同时可对其增删改查。
- 每个定义的实体的名称都为一，可供命令模板配置选择。
- 在每个实体下，可进行实体内容的扩展，即一对多的关系。
- RASA意图模型需要的数据结构分析
- RASA意图训练的数据结构
```
[
{
"intent":"request_query_traffic",
"entities":{
"city": ["广州","深圳","东莞"],
"highway":["京港澳高速","龙华高速"]
},
"template":[
"看一下<city><highway>堵不堵"
"看一下<city><highway>段路况怎么样"
]
}
]
```
- 分析
- 实体部分主要体现在`entities`上。
- 每个`entities`包含了训练模型需要的原始数据，比如实体名称以及其所包含的内容条目。
#### 2. 命令模板分析
- 产品设计分析
- 命令模板包括命令模板名称、引用使用实体数量、实体自定义别名、命令扩展列表。
- 命令模板名称是区别命令模板的唯一性，即一对一的对应关系。
- 一个命令模板可以匹配多个实体，即1对N的对应关系。
- 引用的实体可以用在这个命令模板中自定义唯一性的别名替代，以便在命令扩展列表区分使用。
- 一个命令模板可以匹配多条扩展命令，即1对N的对应关系。
- RASA意图模型需要的数据结构分析
- RASA意图训练的数据结构
```
[
{
"intent":"request_query_traffic",
"entities":{
"city": ["广州","深圳","东莞"],
"highway":["京港澳高速","龙华高速"]
},
"template":[
"看一下<city><highway>堵不堵"
"看一下<city><highway>段路况怎么样"
]
}
]
```
- 分析
- 此结构的训练数据最好划分成三个部分分别存储。
- 第一部分命令模板基本结构。
- 第二部分是命令模板名称与实体匹配结构。
- 第三部分是命令模板名称与实体扩展条目匹配结构
### 数据存储结构SQL设计
#### 1. 实体模块SQL
- 实体主体结构
```
CREATE TABLE `entity`
(
`id` INT(16) PRIMARY KEY AUTO_INCREMENT COMMENT '自增长ID',
`entity_id` VARCHAR(64) UNIQUE NOT NULL COMMENT '实体ID',
`entity_name` VARCHAR(128) UNIQUE NOT NULL COMMENT '实体ID',
`entity_number` int(8) NOT NULL COMMENT '实体条目数量',
`platform` VARCHAR(32) NOT NULL DEFAULT 'voice' COMMENT '使用平台',
`created_at` TIMESTAMP DEFAULT now() COMMENT '创建时间',
`updated_at` TIMESTAMP DEFAULT now() COMMENT '创建时间'
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4;
```
- 实体条目内容，和实体主体结构是一对多的关联关心，关联字段是`entity_id`
```
CREATE TABLE `entity_items`
(
`id` INT(16) PRIMARY KEY AUTO_INCREMENT COMMENT '自增长ID',
`entity_id` VARCHAR(64) NOT NULL COMMENT '实体ID，和`entity`的`entity_id`关联',
`entity_item_index` int(8) NOT NULL COMMENT '实体内容条目的索引',
`entity_item` VARCHAR(128) NOT NULL COMMENT '实体内容条目',
`created_at` TIMESTAMP DEFAULT now() COMMENT '创建时间',
`updated_at` TIMESTAMP DEFAULT now() COMMENT '创建时间'
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4;
```
#### 2. 命令模板SQL
- 命令模板结构
```
CREATE TABLE `command_structure`
(
`id` INT(16) PRIMARY KEY AUTO_INCREMENT COMMENT '自增长ID',
`command_intent_id` VARCHAR(64) UNIQUE NOT NULL COMMENT '命令意图模板ID',
`command_intent_name` VARCHAR(128) UNIQUE NOT NULL COMMENT '命令意图模板名称',
`command_entity_number` int(2) NOT NULL COMMENT '命令实体项中条目数',
`platform` VARCHAR(32) NOT NULL DEFAULT 'voice' COMMENT '使用平台',
`created_at` TIMESTAMP DEFAULT now() COMMENT '创建时间',
`updated_at` TIMESTAMP DEFAULT now() COMMENT '创建时间'
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4;
```
- 命令实体项结构
```
CREATE TABLE `command_entity`
(
`id` INT(16) PRIMARY KEY AUTO_INCREMENT COMMENT '自增长ID',
`command_intent_id` VARCHAR(64) NOT NULL COMMENT '命令意图模板ID，和`command_structure`的`command_intent_id`关联',
`command_entity_tag` VARCHAR(128) NOT NULL COMMENT '命令实体项条目的标识',
`entity_id` VARCHAR(64) NOT NULL COMMENT '实体ID，和`entity`的`entity_id`关联',
`created_at` TIMESTAMP DEFAULT now() COMMENT '创建时间',
`updated_at` TIMESTAMP DEFAULT now() COMMENT '创建时间'
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4;
```
- 命令样板项结构
```
CREATE TABLE `command_template`
(
`id` INT(16) PRIMARY KEY AUTO_INCREMENT COMMENT '自增长ID',
`command_intent_id` VARCHAR(64) NOT NULL COMMENT '命令意图模板ID，和`command_structure`的`command_intent_id`关联',
`command_template_index` int(2) NOT NULL COMMENT '命令样板项索引',
`command_template_item` VARCHAR(512) NOT NULL COMMENT '命令样板项内容',
`created_at` TIMESTAMP DEFAULT now() COMMENT '创建时间',
`updated_at` TIMESTAMP DEFAULT now() COMMENT '创建时间'
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4;
```
### 注意
#### 1.这个设计是配合RASA意图训练使用的，即储存客户自定义的意图训练数据，如果项目中没有使用此种需求，此存储设计没必要部署上去。
#### 2.可能需要和TOKEN鉴权API系统已通过使用。
#### 3.所有的表都建立在同意数据库中，不能分库存储。
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment