如果是VCK190 ES單板，需要在Lounge里申請"Versal Tools Early Eacess"; "Versal Tools PDI Early Eacess"的License，并在Vivado里使能ES器件。在Vivado/2020.2/scripts/init.tcl的文件里，添加“enable_beta_device xcvc*”，可以自動使能ES器件。

1.2. Platform
在進行開發之前，需要準備Platform。 VCK190 Production單板和VCK190 ES單板使用的Platform不一樣，可以從下面鏈接下載各自的Platform，再復制到目錄“Xilinx/Vitis/2020.2/platforms/”下。

VCK190 Production Platform
VCK190 ES Platform

準備好后，目錄結構與下面類似。

1.3. Common Images

Xilinx現在還提供了Common Images，包含對應單板的Linux啟動文件，和編譯器、sysroots（頭文件、應用程序庫）等。可以在Xilinx Download下載Versal common image。

1.4. 測試環境
Host OS： Ubuntu 18.04
Vitis 2020.2
PetaLinux 2020.2
VCK190 Production

2. AIE a2z 分析
2.1. 介紹

例程AIE a2z是Standalone （BareMetal）的例程，Versal的A72不運行Linux。 它很全面，包含創建Platform、創建AIE Kernel、創建PL Kernel、創建A72應用程序、調試AIE Kernel。在Xilinx的文檔中，AIE的程序，叫Kernel； 在Vitis里使用HLS開發的PL設計，也叫Kernel。

注意，2021年7月份，Vitis Tutorials的"master"分支，才包含例程AIE a2z。

2.2. 文件列表
AIE a2z包含下列文件。

后綴是md的文件，詳細說了完成AIE a2z例子各個步驟。

AI_Engine_Development/Feature_Tutorials/01-aie_a_to_z$ ls -l
01-custom_base_platform_creation.md
02-aie_application_creation.md
03-pl_application_creation.md
04-ps_application_creation_run_all.md
images
LICENSE.txt
README.md
script
src

src目錄包含源代碼文件，分別對應A72、PL、AIE設計。
AI_Engine_Development/Feature_Tutorials/01-aie_a_to_z/src$ tree
main.cpp
mm2s.cpp
platform_config.h
platform.cpp
platform.h
s2mm.cpp

2.3. main.cpp
main.cpp是A72的主要代碼。除去Standalone軟件的系統初始化，它還完成以下工作。

1. 分別申請內存，初始化輸入輸出數據
2. 通過AIE_systemConfig啟動MM2S/S2MM的數據搬移
3. 初始化和運行Graph
4. 檢查輸出數據是否正確

int main()
{

printf("Initializing input data in memory\n");
status = InitInputData(&in, INPUT_SIZE);

printf("Initializing output memory space\n");
status = InitOutputData(&out, OUTPUT_SIZE);

printf("System Configuration\n");
AIE_systemConfig(in, out, INPUT_SIZE, OUTPUT_SIZE);

printf("Graph Initialization\n");
mygraph.init();

printf("Running Graph for 4 iterations\n");
mygraph.run(4);

int checks = 1;

while(1) {
printf("Checking output, check #%d\n",checks);
uint32_t v = Xil_In32(S2MM_BASE + CTRL_OFFSET);
checks++;
if(v & 6) {
break;
}
sleep(2);
}

printf("Checking Output Data: \n");
int err = 0;
int32_t golden, dataIn, realTemp, imTemp;
for(int i = 0; i dataIn = ((int32_t*)in)[i];
realTemp = (dataIn & 0xFFFF) + ((dataIn & 0xFFFF0000)>>16);
imTemp = (dataIn & 0xFFFF) - ((dataIn & 0xFFFF0000)>>16);
golden = ((realTemp - imTemp)

if(golden != ((int32_t*)out)[i])
{
err++;
printf("Output Error: Golden = %x Output = %x\n ",golden,((int32_t*)out)[i]);
}
}
}

2.4. mm2s.cpp
mm2s.cpp是利用HLS做的PL設計，用于從內存搬移數據到AIE Kernel。
extern "C" {
void mm2s(ap_int* mem, hls::stream >& s, int size) {
#pragma HLS INTERFACE m_axi port=mem offset=slave bundle=gmem

#pragma HLS interface axis port=s

#pragma HLS INTERFACE s_axilite port=mem bundle=control
#pragma HLS INTERFACE s_axilite port=size bundle=control
#pragma HLS interface s_axilite port=return bundle=control

for(int i = 0; i #pragma HLS PIPELINE II=1
qdma_axis x;
x.data = mem[i];
x.keep_all();
s.write(x);
}
}
}

2.5. s2mm.cpp
mm2s.cpp也是利用HLS做的PL設計，用于從AIE Kernel搬移數據到內存。
extern "C" {

void s2mm(ap_int* mem, hls::stream >& s, int size) {
#pragma HLS INTERFACE m_axi port=mem offset=slave bundle=gmem

#pragma HLS interface axis port=s

#pragma HLS INTERFACE s_axilite port=mem bundle=control
#pragma HLS INTERFACE s_axilite port=size bundle=control
#pragma HLS interface s_axilite port=return bundle=control

for(int i = 0; i #pragma HLS PIPELINE II=1
qdma_axis x = s.read();
mem[i] = x.data;
}
}
}

2.6. AIE工程文件
AIE a2z例子中，沒有提供AIE的代碼。AIE代碼，來自于Vitis中AIE的模板工程“Simple”。AIE的模板工程“Simple”包含文件kernels.h、kernels.cc、project.h、project.cpp。

kernels.cc是定義AIE Kernel的文件，也是最重要的文件，它實際運行在AIE上。

void simple(input_window_cint16 * in, output_window_cint16 * out) {
cint16 c1, c2;
for (unsigned i=0; i window_readincr(in, c1);
c2.real = c1.real+c1.imag;
c2.imag = c1.real-c1.imag;
window_writeincr(out, c2);
}
}

kernels.h聲明了AIE Kernel函數的的原型。

void simple(input_window_cint16 * in, output_window_cint16 * out);

project.h定義了運算的graph，連接了stream數據流和AIE kernel。

class simpleGraph : public adf::graph {
private:
kernel first;
kernel second;
public:
input_port in;
output_port out;
simpleGraph(){

first = kernel::create(simple);
second = kernel::create(simple);
connect > net0 (in, first.in[0]);
connect > net1 (first.out[0], second.in[0]);
connect > net2 (second.out[0], out);

source(first) = "kernels/kernels.cc";
source(second) = "kernels/kernels.cc";

runtime(first) = 0.1;
runtime(second) = 0.1;

}
};

project.cpp定義和控制運算的graph。A72的應用程序，包含了project.cpp，并且執行了mygraph.init()和mygraph.run( )。

simpleGraph mygraph;

# if 0
simulation::platform platform("data/input.txt", "data/output.txt");
# else
PLIO *in0 = new PLIO("DataIn1", adf::plio_32_bits,"data/input.txt");
PLIO *out0 = new PLIO("DataOut1",adf::plio_32_bits, "data/output.txt");
simulation::platform platform(in0, out0);
# endif

connect net0(platform.src[0], mygraph.in);
connect net1(mygraph.out, platform.sink[0]);

# ifdef __AIESIM__

int main(void) {
mygraph.init();
mygraph.run(4);
mygraph.end();
return 0;
}
# endif // __AIESIM__

3. 經驗

AIE a2z做得相當完善，基本可以順利完成。 在實驗過程中，可能遇到下列問題。

3.1. AXI Interrupt
創建平臺（Platform）時，AXI中斷控制器（axi_intc）沒有連接中斷源。Vitis編譯工程時，會連接HLS設計的IP模塊的中斷輸出到AXI中斷控制器（axi_intc）。 如果驗證平臺（Platform）的Block Design時，Vivado會報告下列關于中斷控制器消息，提示沒有中斷源，可以忽略。

[BD 41-759] The input pins (listed below) are either not connected or do not have a source port, and they don't have a tie-off specified. These pins are tied-off to all 0's to avoid error in Implementation flow.
Please check your design and connect them as needed:
/axi_intc_0/intr

3.2. sys_clk0
Vivado也會對輸入時鐘報告下列時鐘不匹配的消息。Vivado創建Block Design時，默認的時鐘是100MHz。單板上的實際時鐘是200MHz。選中sys_clk0_0，在屬性中，把它更改為200MHz。

[xilinx.com:ip:axi_noc:1.0-1] /ps_nocClock frequency of the connected clock (/ps_noc/sys_clk0) is 100.000 MHz while "Input System Clock Frequency" is 200.000 MHz. Please either reconfigure the parameter "Input System Clock Period" of the axi_noc (in DDR Basic tab) or change frequency of the connected clock (CONFIG.FREQ_HZ) within the range of 199920031.987 to 200080032.013 Hz.

3.3. AIE license
如果Vitis編譯工程時，報告“AIE license not found”，請申請license。

AIE license not found !
/opt/Xilinx/Vitis/2020.2/aietools/bin/aieir_be: line 96: kill: (-28000) - No such process
ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> aieir_be --time-passes=0 --trace-plio-width=64 --pl-freq=0 --use-real-noc=true --show-loggers=false --high-performance=false --kernel-address-location=false --target=x86sim --swfifo-threshold=40 --single-mm2s-channel=false --workdir=./Work --exit-after=complete --event-trace-config= --test-iterations=-1 --stacksize=1024 --platform=/proj/hankf/vck190/vck190_aie_a2z/vitis/base_pfm_vck190_aie_a2z/export/base_pfm_vck190_aie_a2z/base_pfm_vck190_aie_a2z.xpfm --event-trace-custom-config= --disable-dma-cmd-alignment=false --enable-ecc-scrubbing=false --write-partitioned-file=true --schemafile=AIEGraphSchema.json --includ --includ --includ --includ --includ --includ --device= --write-unified-data=false --fastmath=false --event-trace-advanced-mapping=0 --log-level=1 --enable-reconfig=false --aiesim-xrt-api=false --gen-graph-cleanup=false --use-canonical-net-names=false --event-trace-port=plio --new-placer=true --use-phy-shim=true --xlopt=0 --pre-compile-kernels=false --validate-only=false --trace-aiesim-option=0 --aiearch=aie --mapped-soln-udm= --optimize-pktids=false --no-init=false --num-trace-streams=1 --aie-heat-map=false --phydevice= --exec-timed=0 --pl-auto-restart=false --routed-soln-udm= --enable-profiling=false --disable-transform-merge-broadcast=false --verbose=true --use-async-rtp-locks=true --repo-path= --genArchive=false --pl-axi-lite=false --new-router=true --aie-driver-v1=false --logcfg-file= --event-trace-bounding-box= --enable-reconfig-dma-autostart=false --heapsize=1024 --logical-arch= --nodot-graph=false --shim-constraints= --disable-dma-autostart=false --disable-transform-broadcast-split=true -json ./Work/temp/project.json -sdf-graph /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.cpp.

3.4. 安裝dot
Vitis在編譯過程中，會用到工具dot。如果沒有安裝sudo apt install graphviz，會得到錯誤"sh: 1: dot: not foun"。 在Ubuntu 18.04下，如果有管理員權限，使用命令“sudo apt install graphviz”能安裝dot。

DEBUG:MapperPartitioner: Adding Edge : Name=D_net2 SrcPort=i1_po0 DstPort=i3_pi0 EdgeType=mem
DEBUG:MapperPartitioner:Done--Add Double Buffer Edge SrcPort=i1_po0 DstPort=i3_pi0 type=mem Edge=net2:i1-(buf2)->i3
DEBUG:MapperPartitioner:Graph After Adding Double Edges
sh: 1: dot: not found

ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> dot ./Work/reports/project.dot -Tpng -o ./Work/reports/project.png
.
Please check the output log for errors and fix those before you run the application.
/opt/Xilinx/Vitis/2020.2/aietools/bin/aieir_be: line 96: kill: (-44668) - No such process
ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> aieir_be --time-passes=0 --trace-plio-width=64 --pl-freq=0 --use-real-noc=true --show-loggers=false --high-performance=false --kernel-address-location=false --target=hw --swfifo-threshold=40 --single-mm2s-channel=false --workdir=./Work --exit-after=complete --event-trace-config= --test-iterations=-1 --stacksize=1024 --platform=/proj/hankf/vck190/vck190_aie_a2z/vitis/base_pfm_vck190_aie_a2z/export/base_pfm_vck190_aie_a2z/base_pfm_vck190_aie_a2z.xpfm --event-trace-custom-config= --disable-dma-cmd-alignment=false --enable-ecc-scrubbing=false --write-partitioned-file=true --schemafile=AIEGraphSchema.json --includ --includ --includ --includ --includ --includ --device= --write-unified-data=false --fastmath=false --event-trace-advanced-mapping=0 --log-level=1 --enable-reconfig=false --aiesim-xrt-api=false --gen-graph-cleanup=false --use-canonical-net-names=false --event-trace-port=plio --new-placer=true --use-phy-shim=true --xlopt=0 --pre-compile-kernels=false --validate-only=false --trace-aiesim-option=0 --aiearch=aie --mapped-soln-udm= --optimize-pktids=false --no-init=false --num-trace-streams=1 --aie-heat-map=false --phydevice= --exec-timed=0 --pl-auto-restart=false --routed-soln-udm= --enable-profiling=false --disable-transform-merge-broadcast=false --verbose=true --use-async-rtp-locks=true --repo-path= --genArchive=false --pl-axi-lite=false --new-router=true --aie-driver-v1=false --logcfg-file= --event-trace-bounding-box= --enable-reconfig-dma-autostart=false --heapsize=1024 --logical-arch= --nodot-graph=false --shim-constraints= --disable-dma-autostart=false --disable-transform-broadcast-split=true -json ./Work/temp/project.json -sdf-graph /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.cpp.

3.5. 軟件Emulation
運行軟件Emulation的時候，要選擇AIE工程，不選擇system project。如果選擇system project運行軟件Emulation，會出現下列錯誤。

Error while launching program:
The selected system project 'simple_application_system' contains applications (simple_application) that doesn't support launching software emulation.
The selected system project 'simple_application_system' contains applications (simple_application) that doesn't support launching software emulation.

3.6. 硬件Emulation
先運行軟件Emulation，再運行硬件Emulation。
如果直接運行硬件Emulation，會出現下列錯誤。

Failed to start emulator on the project 'simple_application_system' using the build configuration 'Emulation-HW'.
Launch emulator script doesn't exist at location '/proj/hankf/vck190/vck190_aie_a2z_script_hw_prj/custom_pfm_vck190/vitis/simple_application_system/Emulation-HW/package/launch_hw_emu.sh'.

另外Vitis里，先選擇AIE工程，再編譯AIE工程，然后去啟動硬件Emulation，菜單里可能沒有目標。編譯后，要重新選擇system project，再選擇AIE工程，再去啟動硬件Emulation，菜單里就會有目標。

3.7. A72軟件沒有ap_int.h
文件mm2s.cpp和s2mm.cpp時給HLS設計用的，不能添加到A72的軟件工程里。如果把它們加到了A72的軟件工程里，會遇到錯誤“ap_int.h: No such file or directory”。

aarch64-none-elf-g++ -Wall -O0 -g3 -I"/opt/Xilinx/Vitis/2020.2/aietools/include"
-I"/proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src"
-I"/../include" -c -fmessage-length=0 -MT"src/mm2s.o" -mcpu=cortex-a72 -I/proj/hankf/vck190/vck190_aie_a2z/vitis/vck190_aie_a2z_aie_output_platform/export/vck190_aie_a2z_aie_output_platform/sw/vck190_aie_a2z_aie_output_platform/standalone_domain/bspinclude/include -MMD -MP -MF"src/mm2s.d" -MT"src/mm2s.o" -o "src/mm2s.o" "../src/mm2s.cpp"
../src/mm2s.cpp:33:10: fatal error: ap_int.h: No such file or directory
33 | #include
| ^~~~~~~~~~

3.8. A72軟件工程找不到simple(input_window, output_window)
A72軟件要控制AIE Kernel，需要相關信息。因此預先把AIE工程編譯后產生的文件“Hardware/Work/ps/c_rts/aie_control.cpp“，添加到 A72軟件工程。

如果忘記添加，可能會得到錯誤信息，“undefined reference to `simple(input_window, output_window)'”

aarch64-none-elf-g++ -L/opt/Xilinx/Vitis/2020.2/aietools/lib/aarchnone64.o -mcpu=cortex-a72 -Wl,-T -Wl,../src/lscript.ld -L/proj/hankf/vck190/vck190_aie_a2z/vitis/vck190_aie_a2z_aie_output_platform/export/vck190_aie_a2z_aie_output_platform/sw/vck190_aie_a2z_aie_output_platform/standalone_domain/bsplib/lib -o "aie_a2z_vck190_a72_ctrl_app.elf" ./src/main.o ./src/platform.o -ladf_api -Wl,--start-group,-lxil,-lgcc,-lc,-lstdc++,--end-group
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: ./src/main.o: in function `simpleGraph::simpleGraph()':
/proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:17: undefined reference to `simple(input_window*, output_window*)'
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:17: undefined reference to `simple(input_window*, output_window*)'
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:18: undefined reference to `simple(input_window*, output_window*)'
makefile:48: recipe for target 'aie_a2z_vck190_a72_ctrl_app.elf' failed
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:18: undefined reference to `simple(input_window*, output_window*)'
collect2.real: error: ld returned 1 exit status
make: *** [aie_a2z_vck190_a72_ctrl_app.elf] Error 1

3.9. Package

編譯A72程序后，要編譯system project，將所有模塊打包再一起。這時候，要根據04-ps_application_creation_run_all.md的Step 3. Build the Full System，添加打包選項，“--package.ps_elf ../../A-to-Z_app/Debug/A-to-Z_app.elf,a72-0 --package.defer_aie_run”。

如果沒有添加，會報告錯誤“no xclbin input is found”。

Package step cannot be performed since the platform has a VPP link generated XSA and no xclbin input is found. Please provide a valid xclbin location in system project package options
11:21:21 Build Finished (took 646ms)