2023/05/31

mysql explain select

 ## explain select

- access_type:
- all(全件検索)/index(indexの全件検索)はfull-scanであり、要注意。
- indexはallより良い。
- possible_keys: 使用可能なindex(primary-key is index)
- key: 実際使ったindex
- ref: indexのKeyと比較するcolumn
- patition: 使ったpatition. patationは何?大きなテーブルを分解すること
- patitionは使えるかも!
- 分解のtypeは:RANGE,LIST,HASH,KEY
- rows: 大体の件数を推測した結果
- filtered: 条件が効いて、次の層に行く件数の割合。小さい方が良い。

2023/05/26

rails,where or

 

  1. def check
  2. @spec = Hash.new
  3. @spec = Spec.merge(Spec.where(a:1).or(Spec.where(b:0)))
  4. .merge(Spec.where(c:0).or(Spec.where(d:0))).first
  5. end

2023/05/17

change git http to ssh

 - git add ssh key
        - ssh-keygen -t rsa 
        - update to github ssh key
        - 
ssh -T git@github.com

     - add to .ssh/config

Host github github.com HostName github.com IdentityFile ~/.ssh/id_git_rsa #ここに自分の鍵のファイル名 User git

ssh -T github



- change git repository from http to ssh

    git remote -v

    git remote set-url origin git@github.com:xx/xx.git

git remote -v

2023/01/29

mysql lock

 - mysql lock 

  - lock mode
- share lock , S, for read
- exclusive lock, X,for update or deleteeven read have to wait the lock.
- lock granularity(粒度)
- table lock
- row lock(record lock)
- A record lock is a lock on an index record.
- if no index, innodb will create a hidden cluster index to lock.
- show engine innodb status;
```
RECORD LOCKS space id 58 page no 3 n bits 72 index `PRIMARY` of table `test`.`t`
trx id 10078 lock_mode X locks rec but not gap
Record lock, heap no 2 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
```
- gap lock(range lock)
- lock a gap between index records
- for example, select ... where id between 1 and 10 for update.
- Gap locking is not needed for statements that lock rows using a unique index to search for a unique row
- only purpose is to prevent other transactions from inserting to the gap (Gap内にInsertできない)
- can co-exist. does not prevent another transaction from taking a gap lock on the same gap
- 最初のインデックスレコードの前または最後のインデックスレコードの後のギャップのロック
- (Gap内にInsertできない)UpdateDeleteできる!!
- Next Key lock
- a combination of
- a record lock on the index record
- and a gap lock on the gap before the index record.
- If one session has a shared or exclusive lock on record R in an index, another session cannot insert a new index record in the gap immediately before R in the index order
- show engine innodb status
```
RECORD LOCKS space id 58 page no 3 n bits 72 index `PRIMARY` of table `test`.`t`
rx id 10080 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
```
- Insert Intention Locks
- a type of gap lock set by INSERT operations prior to row insertion
- multiple transactions inserting into the same index gap need not wait for each other if they are not inserting at the same position within the gap
- AUTO-INC Locks
- transactions inserting into tables with AUTO_INCREMENT columns

- intention lock, multiple granularity lock.(粒度が多いlock),lock modelock targetの組み合わせ
- intention lock is table level lock that show the row lock a transaction will want on a row
- テーブル級のロックであるが、次のTXの行ロックを提示する
- intention share lock,IS. a tx intents to set a share lock on a row in a table.
- for example. select *** for share
- intention exclusive lock,IX. a tx intents to set a exclusive lock on a row in a table.
- for example. select *** for update
- Before a transaction can acquire a shared lock on a row in a table, it must first acquire an IS lock or stronger on the table.
- Before a transaction can acquire an exclusive lock on a row in a table, it must first acquire an IX lock on the table.
- Intention locks do not block anything,
- The main purpose of intention locks is to show that someone is locking a row, or going to lock a row in the table.
- ロック発生前に予測してErrorにする、実際のロックを回避する?
- show engine innodb status; TABLE LOCK table `test`.`t` trx id 10080 lock mode IX

example:
```

show engine innodb status;

------------------------
LATEST DETECTED DEADLOCK
------------------------
2023-01-27 10:24:38 0x409e8ed700
*** (1) TRANSACTION:
TRANSACTION 22982, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 13158, OS thread handle 277537257216, query id 175713 172.18.0.1 root update
insert into

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 70 page no 3 n bits 264 index PRIMARY of table `w`.`t` trx id 22982 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;

*** (2) TRANSACTION:
TRANSACTION 22981, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1136, 2 row lock(s)
insert into
MySQL thread id 13157, OS thread handle 277538068224, query id 175714 172.18.0.1 root update

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 70 page no 3 n bits 264 index PRIMARY of table `w`.`t` trx id 22981 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 70 page no 3 n bits 264 index PRIMARY of table `w`.`t` trx id 22981 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;

============================
```

```
In my circumstance, the only index is PRIMARY, which was useless in my SELECT query (can be verified by explain select...).
As a result, all entries in the PRIMARY index were locked.
When TXN_2 waited for an X lock on a certain entry, the entry was locked by an S lock retained by TXN_1.
Similarly, TXN_1 waited for an X lock on another entry but the entry was also locked by S lock retained by itself.
A "one S two X" deadlock occurred.

By contrast, after I created an index name on the column name,
the index name would be used in the SELECT statement (can be verified by explain select ...),
so the locks would be issued on the index name instead of PRIMARY.
More importantly, the SELECT statement would only issue S lock on the entry equals to someValue instead of all entries of index name.
Besides, the IX lock and X lock required by INSERT would be issued on the index PRIMARY.
Conflicts between S lock and IX lock, X lock would be solved.

The index on column name did not only speed up the query but more importantly prevented locking all entries of the index.

```

2022/12/15

golang option pattern

 - golang option pattern

- option struct
```
type option struct {
Name string
Age int
}
```
- option func type
```
type Option func(*options)

func F1(name string) Option{
return fun(op *option){
op.Name = name
}
}
}

func F2(age int) Option{
return func(op *option) {
op.Age= age
}
}
```
- usage
```
func NewOption(setters ...Option){
op := &option{}
for s:=range setters{
s(op)
}
return op
}

NewOption(F1("tom"),F2(18))
```

2022/12/14

what is good bulk insert

To get optimal write throughput for bulk loads, partition your data by primary key with this pattern:
  Each partition contains a range of consecutive rows. Each commit contains data for only a single partition. A good rule of thumb for your number of partitions is 10 times the number of nodes in your Cloud Spanner instance. So if you have N nodes, with a total of 10*N partitions, you can assign rows to partitions by:

  Sorting your data by primary key. Dividing it into 10*N separate sections. Creating a set of worker tasks that upload the data. Each worker will write to a single partition. Within the partition, it is recommended that your worker write the rows sequentially. However, writing data randomly within a partition should also provide reasonably high throughput.

  As more of your data is uploaded, Cloud Spanner automatically splits and rebalances your data to balance load on the nodes in your instance. During this process, you may experience temporary drops in throughput.

  Following this pattern, you should see a maximum overall bulk write throughput of 10-20 MiB per second per node.

golang chunk

 - chunk

```
func Chunk[T any](items []T, chunkSize int) [][]T {
var chunks = make([][]T, 0, (len(items)/chunkSize)+1)
for chunkSize < len(items) {
items, chunks = items[chunkSize:], append(chunks, items[0:chunkSize:chunkSize])
}
return append(chunks, items)
}
//Arrayをスライシングすると、リターンされたSliceのcapはstartIndexからArrayの最期までのサイズです。
//しかし、次のようにスライシングする時、maxIndexを追加してcapのサイズを調整することが出来ます。
//slice[startIndex:endIndex:maxIndex]
```

big array goroutine

 - https://qiita.com/tutuz/items/057452fbbe9a5ae26d37

- multi goroutine to loop array
```
slice := []string{"a", "b", "c", "d", "e"}
sliceLength := len(slice)
var wg sync.WaitGroup
wg.Add(sliceLength)
fmt.Println("Running for loop…")
for i := 0; i < sliceLength; i++ {
go func(i int) {
defer wg.Done()
val := slice[i]
fmt.Printf("i: %v, val: %v\n", i, val)
}(i)
}
wg.Wait()
fmt.Println("Finished for loop")
```
- 課題はエラー検知とgoroutineの数の制御、errgroupの登場
- golang.org/x/sync/errgroup は、複数の goroutine を実行して、それらのうちにエラーがあったときにエラーを知る、ということを可能にしてくれるライブラリ
- sync.WaitGroup は実行した goroutine が終わるのを待ちますが、エラーがあったかどうかはわかりません
- errgroup は sync.WaitGroup+error といったイメージで、どれかの goroutine でエラーがあったら最初のひとつを知ることができ
```
// You can edit this code!
// Click here and start typing.
package main

import (
"context"
"fmt"

"golang.org/x/sync/errgroup"
"golang.org/x/sync/semaphore"
)

func main() {
fmt.Println("Hello, 世界")
ctx := context.TODO()
ms := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
mcs := make([]int, len(ms))

eg := errgroup.Group{}
sem := semaphore.NewWeighted(3)
for i := range ms {
if err := sem.Acquire(ctx, 1); err != nil {
panic(err)
}
func(i int) {
eg.Go(
func() error {
defer sem.Release(1)
fmt.Printf("i is %v\n", i)
mcs[i] = ms[i] + 100
return nil
})
}(i)
}

if err := eg.Wait(); err != nil {
panic(err)
}
fmt.Printf("ms is %v\n", ms)
fmt.Printf("msc is %v\n", mcs)
}
```