Assignment # 02
Distribution date: 20.09.13 0000 HRS Due date: 24.02.14 2359 HRS
Note Carefully:
Ø Read and understand the Course Policy already circulated on the group and available at https://groups.yahoo.com/neo/groups/SMPAN-mhi/files/ i.e. the file section of the group.
Ø Students are only allowed to submit assignment through email.
Ø Acceptable formats are MS Word (doc / docx) ONLY.
Ø Submission deadline through email is 11:59 PM
Ø Copy paste in answer to any question from other assignment is not allowed. It MAY result in zero credit for the assignment. Give answers to questions in your own words. Give complete reference of your data.
Ø File name SHOULD BE in all capital letter and in the following format CSS-ASG-01-YourRollNumber.doc. e.g.
ü CSS-ASG-01-FA-2013-MSC-CE-001.DOC or
ü CSS-ASG-01-SP-2013-MS-IS-001.DOC etc
Ø The subject field of your email MUST BE “CSS-ASG-01-YourRollNumber-YourName" with no spaces. e.g.
ü CSS-ASG-01-FA-2013-MSC-CE-001-WAQARKHAN or
ü CSS-ASG-01-SP-2013-MS-IS-001-NASIRALI etc
Ø Name your files and subject your email correctly as they will be handled by auto filters.
Ø Send your e-mail to the teaching assistant (saqi.17@gmail.com or qasim_basit2000@yahoo.com), no assignment will be entertained on group id.
Ø Late assignments will NOT be accepted, what so ever is the reason.
Ø The submission deadline will not be extended in any case.
ENCLOSE THE FOLLOWING INFORMAITON AT THE START OF YOUR ASSIGNMENT.
CSS ASSIGNMENT NUMBER: _________________________________________________
STUDENT ROLL NUMBER: ___________________________________________________
STUDENT NAME: ____________________________________________________________
ASSIGNMENT # 02
Question No. 01 (20)
Select an area of computer systems (for example, processor design, networks, operating systems, or databases), review articles on performance evaluation in that area and make a list of benchmarks used in those articles.
Area Selected : Android Systems Performance Evaluation:
Articles Reviewed:
Performance Analysis of Android Underlying Virtual Machine in Mobile Phones
Evaluating Performance of Android Platform Using Native C for
Embedded Systems
Measuring and Improving Application Launching Performance on Android Devices
Analysis of Android Malware Detection Performance using Machine Learning Classifiers
Sleeping Android:Exploit through Dormant Permission Requests
Question No. 02 (30)
Make a complete list of metrics to compare
Ø Two personal computers
Ø Two database systems
Ø Two disk drives
Ø Two window systems
Best of luck
CSS ASSIGNMENT NUMBER: ___________2___________________________
STUDENT ROLL NUMBER: ___Sp-2014-MSC-CE 011___________________
STUDENT NAME: ______________Kashif Islam ________________________
Area Selected: Android Systems Performance Evaluation
Articles Reviewed:
- Performance Analysis of Android Underlying Virtual Machine in Mobile Phones
- Catch Me if You Can Evaluating Android Anti-malware against Transformation Attacks
- Evaluating Performance of Android Platform Using Native C for Embedded Systems
- Sleeping Android: Exploit through Dormant Permission Requests
- Measuring and Improving Application Launching Performance on Android Devices
- Analysis of Android Malware Detection Performance using Machine Learning Classifiers
- Enhancing Performance of Traffic Safety Guardian System on Android by Task Skipping Mechanism
- Performance Analysis of Android Underlying Virtual Machine in Mobile Phones
SUT: Android
CUS: DVM
Analysis Method: Measurement and simulation
Profiling is not available in android so to measure performance “Debug” and “VmDebug” default libraries are used.
Benchmark used:
Finding a benchmark was a challenge here.
Benchmark uses should also be using DVM.
Benchmark chosen should have its source code available.
So from Android SDK seven common and popular applications were chosen and three default android applications were chosen which include Camera, Music Player and Calculator as a benchmark for comparing performance of DVM.
- Catch Me if You Can Evaluating Android Anti-malware against Transformation Attacks
This is most interesting paper for evaluating Anti-malware software performance on android.
SUT: Android Anti malware softwares
CUS: Transformed Anti Malwares
Analysis Method: Measurement and simulation
Benchmark used:
Malware detection against following transformation metrics:
- Evaluating Performance of Android Platform Using Native C for Embedded Systems
SUT: NDK
CUS: JNI vs native JAVA
Analysis Method: Measurement only
Benchmark used: JNI was used as benchmark in this study and native Java performance was measured against it.
Performance metrics:
- JNI communication delay
- Integer Calculation
- Floating Point Calculation
- Memory Access Algorithms
- Heap Memory Allocation Algorithms
Results show that native C/C++ used through JNI has better performance against all metrics except for memory access algorithms.
In Memory Access Native Java performs better than JNI in android.
- Measuring and Improving Application Launching Performance on Android Devices
SUT: Android Applications
CUS: Launching Speed and Performance
Analysis Method: Measurement and Simulation
Performance Metric: Launch time
Benchmark used: Default preload and no preload of classes were used as a benchmark against +51 classes , +120 classes and +281 classes.
Results:
- Analysis of Android Malware Detection Performance using Machine Learning Classifiers
SUT: Android Malwares and their detection using MLCs
CUS: Machine Level Classifiers Performance
Analysis Method: Measurement and Simulation
Performance Metric: True Positive Rate, False Positive Rate,Precision,
Workload : Malwares GoldDream, PJApps, DroidKungFu2, Snake, Angry Birds Rio
Unlocker
Benchmark selected: Naïve Bayesian, RandomForest, Logistic Regression, SVM: Support Vector Machine
Result : Shown interms of confusion matrix.
- Enhancing Performance of Traffic Safety Guardian System on Android by Task Skipping Mechanism
SUT: Traffoc Safety Guardian System on Android
CUS: TSG frames
Metrics: Car detection speed and fps
Workload: moving cars and highway lines
Benchmark: original fps without Task Skipping
Result: increased fps by using task skipping and JNI
Conclusion
Performance comparison in android devices still lacks analytical method. Mostly performance analysis is conducted using measurement or simulation which is time consuming. Benchmarks used for performance analysis are native implementations without suggested improvements and these are then compared with results of improvement implemented schemes / apps.
About the Author.
8 years’ experience in Nokia Siemens Networks and Ericsson, Intelligent Networks and Charging Systems
Humanist, philanthropist and Technologist
References
- Performance Analysis of Android Underlying Virtual Machine in Mobile Phones ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6336470
- Catch Me if You Can Evaluating Android Anti-malware against Transformation Attacks
ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6661334
- Evaluating Performance of Android Platform Using Native C for Embedded Systems
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5669738
- Sleeping Android: Exploit through Dormant Permission Requests
www.ma.rhul.ac.uk/static/techrep/2013/MA-2013-06.pdf
- Measuring and Improving Application Launching Performance on Android Devices
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6726978
- Analysis of Android Malware Detection Performance using Machine Learning Classifiers
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6675404
- Enhancing Performance of Traffic Safety Guardian System on Android by Task Skipping Mechanism
ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6570137
Question No. 02 (30)
Make a complete list of metrics to compare
- Two personal computers
- Two database systems
- Two disk drives
- Two window systems
- Performance metrics to compare Two personal computers
Throughput: Processing speed, Memory speed, bus speed.
Response time: boot time, disks RPM / IO speed battery Discharge Curve
Resource: Disk and memory capacity, number of ports. Display technology used (LCD or LED)
In addition to above following are performance metrics of individual components for comparing two PCs
CPU performance metrics: clocking speed,L1/L2/L3 cache size
Graphics card performance metrics: Graphics Processing Clusters, Streaming Multiprocessors,CUDA Cores, Texture Units,ROP Units, Base Clock, Boost Clock, Memory Clock (Data rate),L2 Cache Size, Total Video Memory, Memory Interface, Total Memory Bandwidth, Texture Filtering Rate (Bilinear),Fabrication Process used, Transistor Count, Connectors used, Form Factor, Power Connectors quality, Thermal Design Power (TDP),Thermal Threshold
- Metrics to compare Two database systems
Throughput: TPS, w/s ,r/
Response time: indexing time, query time, loading time, database lock time, measuring concurrent operations and lock percentage. Crash recovery time , connection time, Deadlock discovery and resolution time
Resource Resources consumed in Journaling, processing overhead, I/O on underlying disks (supported block size), load on db server, load on network
In addition to above following performance metrics can be used depending upon SUT
Schema comparison
Timeliness and freshness of data metrics under multiple load conditions.
I plan to compare Berkeley DB, MongoDB and oracle in future provided research yields financial benefits
- Metrics to compare Two disk drives
Throughput: Short stroking
Response time: Seek time, Data transfer rate, Media rate, Sector overhead time, Head switch time, Cylinder switch time,
Resource: power consumption,
Magnetic material lifetime, size, weight, shock absorbent quotient value
Audible noise, Shock resistance,
- Performance Metrics to compare Two window systems
Startup time
Backwards compatibility, Software compatibility,
File copy operations (newer windows system is more optimized)
CPU Usage,cores,
Memory management. Display, graphics
Security, Data execution prevention, Netbook support
Availability of official support, BranchCache support (new feature to speed network ,absent in older versions)
Minimum Resources constraint ( e.g older windows could run on lower slower processors and does not require much RAM)
Kernel type (hybrid or native)
Platform support (supports both 32 and 64 architectures or only one of them)
Physical Memory Limits
About the Author.
8 years’ experience in Nokia Siemens Networks and Ericsson, Intelligent Networks and Charging Systems
Humanist, philanthropist and Technologist
